Fedora / GitLab SSH Denied, Help Understanding Verbose Logs












0















My ssh connection to gitlab fails on my fedora 28 box, with no password request needed. This is preventing me from successfully running needed git commands. I've researched this thoroughly within SO and Googs, but nothing I found seems to work for my issue, more importantly, help explain the various sections of a 'vvv' log.



Set Up




Selinux = disabled



SSH directory: /home/justin/.ssh



    drwx------   2 justin   root  4096 Nov 19 22:16  .ssh


Step 1) Create the keys: ssh-keygen -t rsa -b 4096 -C "j.r.schwimmer@domain.com"



I accept the default location and then enter a pass phrase creating:



    -rw------- 1 justin root 3326 Nov 19 22:16 id_rsa

-rw-r--r-- 1 justin root 749 Nov 19 22:16 id_rsa.pub


Step 2) Grab the contents of id_rsa.pub and add the key to Gitlabs SSH section, verifying that the entire public key is entered into Gitlabs form.



Step 3) Verify that the ssh-agent is running: eval "$(ssh-agent -s)"



Step 4) Add the private key: ssh-add /home/justin/.ssh/id_rsa



Step 5) Run my ssh command: ssh -vvvT justin@gitlab.domainname.net
Confirming that the connection should create a known hosts file



Step 6) Cry after seeing: justin@gitlab.domainname.net: Permission denied (publickey)




The Verbose (vvv) Log



Loads config settings including those found in gitlab.conf:



OpenSSH_7.7p1, OpenSSL 1.1.0h-fips  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 2: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/gitlab.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/gitlab.conf


Makes connection:



debug2: resolving "gitlab.domainname.net" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to gitlab.domainname.net [17.17.17.17] port 22.
debug1: Connection established.


Identity File Section...(Not sure what its doing?)



debug1: identity file /home/justin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/justin/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/justin/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
...checking others
debug1: identity file /home/justin/.ssh/id_xmss-cert type -1


More logs that appear to be handshake related (I think?) / known_hosts setup:



debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.1
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.domainname.net:22 as 'justin'
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31


Known host check:



debug1: Server host key: ecdsa-sha2-nistp256 SHA256:hyO5F3tsdf34PQGAYYhmoGW5J/leBnC0hj3IoE1F68
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 17.17.17.17
debug1: Host 'gitlab.domainname.net' is known and matches the ECDSA host key.
debug1: Found key in /home/justin/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0


Final Auth Verfication:



debug1: rekey after 4294967296 blocks
debug2: key: /home/justin/.ssh/id_rsa (0x55d580f2caf0), agent
debug2: key: /home/justin/.ssh/id_dsa ((nil))
debug2: key: /home/justin/.ssh/id_ecdsa ((nil))
debug2: key: /home/justin/.ssh/id_ed25519 ((nil))
debug2: key: /home/justin/.ssh/id_xmss ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51


(A) Is the auth section below responsible for the overall failure?
(B) It seems to be trying to load every key type after successfully trying id_rsa, is that correct?



debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/justin/.ssh/id_dsa
debug3: no such identity: /home/justin/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ecdsa
debug3: no such identity: /home/justin/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ed25519
debug3: no such identity: /home/justin/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_xmss
debug3: no such identity: /home/justin/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
justin@gitlab.domainname.net: Permission denied (publickey).


Failed Tests




  • I've tried setting this all up as a root user... failed

  • Changing key files ownership, groups and permissions... failed

  • I've tried different key types (id_ecdsa)... failed


  • I tried adding a config file to /etc/ssh/ssh_config.d/ that contained the following:



    Host gitlab.com
    StrictHostKeyChecking no
    LogLevel VERBOSE



More Questions



(C) debug1: identity file /home/justin/.ssh/id_rsa type 0: Line within the "Identity File Section", is this 0 value supposed to be 2? What does this value even mean, the number of key files it found per type?



(D) I'm unable to find an auth.log (or any logging for ssh) file within: /var/log/, why would that be or how can I fix this (Fedora 28)?



Thanks you for your time!



UPDATED: The Verbose (vvvv) Log



OpenSSH_7.7p1, OpenSSL 1.1.0h-fips  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 2: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/gitlab.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/gitlab.conf
debug1: /etc/ssh/ssh_config.d/gitlab.conf line 2: Applying options for gitlab.domainname.net
debug2: resolving "gitlab.domainname.net" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to gitlab.domainname.net [17.17.17.17] port 22.
debug1: Connection established.
debug1: identity file /home/justin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/justin/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.1
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.domainname.net:22 as 'justin'
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:hyO5F3t2OPc2nPQGAYYhmoGW5J/leBnC0hj3IoE1F68
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 17.17.17.17
debug1: Host 'gitlab.domainname.net' is known and matches the ECDSA host key.
debug1: Found key in /home/justin/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: /home/justin/.ssh/id_rsa (0x55568d375a80), agent
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
justin@gitlab.domainname.net: Permission denied (publickey).









share|improve this question




















  • 1





    You ssh folder and key permission are correct and good. Nothing wrong about that. have you add ssh keys to gitlab.com/profile/keys ? if you self hosted gitlab, find it on setting menu

    – Robbi Nespu
    Nov 24 '18 at 1:20








  • 1





    post full ssh -vvvv git@gitlab.com log. This will helping to identify the problem.

    – Robbi Nespu
    Nov 24 '18 at 2:11











  • Hello @RobbiNespu, thanks for checking out the permissions. I have added the public key to gitlab's site. (it isn't self hosted.) Also thanks for the "-vvvv" tip! I'll review that and post it if your answer below doesn't fix my issues

    – Justin Schwimmer
    Nov 26 '18 at 16:57











  • Just to make sure, can you run restorecon -Rv ~/.ssh and test ssh?

    – Robbi Nespu
    Nov 28 '18 at 2:27











  • Running that command produced no results - no files are listed as being changed, it just gave me a command prompt below. Does it matter that my SELinux is disabled?

    – Justin Schwimmer
    Nov 28 '18 at 6:19
















0















My ssh connection to gitlab fails on my fedora 28 box, with no password request needed. This is preventing me from successfully running needed git commands. I've researched this thoroughly within SO and Googs, but nothing I found seems to work for my issue, more importantly, help explain the various sections of a 'vvv' log.



Set Up




Selinux = disabled



SSH directory: /home/justin/.ssh



    drwx------   2 justin   root  4096 Nov 19 22:16  .ssh


Step 1) Create the keys: ssh-keygen -t rsa -b 4096 -C "j.r.schwimmer@domain.com"



I accept the default location and then enter a pass phrase creating:



    -rw------- 1 justin root 3326 Nov 19 22:16 id_rsa

-rw-r--r-- 1 justin root 749 Nov 19 22:16 id_rsa.pub


Step 2) Grab the contents of id_rsa.pub and add the key to Gitlabs SSH section, verifying that the entire public key is entered into Gitlabs form.



Step 3) Verify that the ssh-agent is running: eval "$(ssh-agent -s)"



Step 4) Add the private key: ssh-add /home/justin/.ssh/id_rsa



Step 5) Run my ssh command: ssh -vvvT justin@gitlab.domainname.net
Confirming that the connection should create a known hosts file



Step 6) Cry after seeing: justin@gitlab.domainname.net: Permission denied (publickey)




The Verbose (vvv) Log



Loads config settings including those found in gitlab.conf:



OpenSSH_7.7p1, OpenSSL 1.1.0h-fips  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 2: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/gitlab.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/gitlab.conf


Makes connection:



debug2: resolving "gitlab.domainname.net" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to gitlab.domainname.net [17.17.17.17] port 22.
debug1: Connection established.


Identity File Section...(Not sure what its doing?)



debug1: identity file /home/justin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/justin/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/justin/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
...checking others
debug1: identity file /home/justin/.ssh/id_xmss-cert type -1


More logs that appear to be handshake related (I think?) / known_hosts setup:



debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.1
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.domainname.net:22 as 'justin'
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31


Known host check:



debug1: Server host key: ecdsa-sha2-nistp256 SHA256:hyO5F3tsdf34PQGAYYhmoGW5J/leBnC0hj3IoE1F68
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 17.17.17.17
debug1: Host 'gitlab.domainname.net' is known and matches the ECDSA host key.
debug1: Found key in /home/justin/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0


Final Auth Verfication:



debug1: rekey after 4294967296 blocks
debug2: key: /home/justin/.ssh/id_rsa (0x55d580f2caf0), agent
debug2: key: /home/justin/.ssh/id_dsa ((nil))
debug2: key: /home/justin/.ssh/id_ecdsa ((nil))
debug2: key: /home/justin/.ssh/id_ed25519 ((nil))
debug2: key: /home/justin/.ssh/id_xmss ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51


(A) Is the auth section below responsible for the overall failure?
(B) It seems to be trying to load every key type after successfully trying id_rsa, is that correct?



debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/justin/.ssh/id_dsa
debug3: no such identity: /home/justin/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ecdsa
debug3: no such identity: /home/justin/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ed25519
debug3: no such identity: /home/justin/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_xmss
debug3: no such identity: /home/justin/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
justin@gitlab.domainname.net: Permission denied (publickey).


Failed Tests




  • I've tried setting this all up as a root user... failed

  • Changing key files ownership, groups and permissions... failed

  • I've tried different key types (id_ecdsa)... failed


  • I tried adding a config file to /etc/ssh/ssh_config.d/ that contained the following:



    Host gitlab.com
    StrictHostKeyChecking no
    LogLevel VERBOSE



More Questions



(C) debug1: identity file /home/justin/.ssh/id_rsa type 0: Line within the "Identity File Section", is this 0 value supposed to be 2? What does this value even mean, the number of key files it found per type?



(D) I'm unable to find an auth.log (or any logging for ssh) file within: /var/log/, why would that be or how can I fix this (Fedora 28)?



Thanks you for your time!



UPDATED: The Verbose (vvvv) Log



OpenSSH_7.7p1, OpenSSL 1.1.0h-fips  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 2: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/gitlab.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/gitlab.conf
debug1: /etc/ssh/ssh_config.d/gitlab.conf line 2: Applying options for gitlab.domainname.net
debug2: resolving "gitlab.domainname.net" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to gitlab.domainname.net [17.17.17.17] port 22.
debug1: Connection established.
debug1: identity file /home/justin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/justin/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.1
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.domainname.net:22 as 'justin'
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:hyO5F3t2OPc2nPQGAYYhmoGW5J/leBnC0hj3IoE1F68
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 17.17.17.17
debug1: Host 'gitlab.domainname.net' is known and matches the ECDSA host key.
debug1: Found key in /home/justin/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: /home/justin/.ssh/id_rsa (0x55568d375a80), agent
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
justin@gitlab.domainname.net: Permission denied (publickey).









share|improve this question




















  • 1





    You ssh folder and key permission are correct and good. Nothing wrong about that. have you add ssh keys to gitlab.com/profile/keys ? if you self hosted gitlab, find it on setting menu

    – Robbi Nespu
    Nov 24 '18 at 1:20








  • 1





    post full ssh -vvvv git@gitlab.com log. This will helping to identify the problem.

    – Robbi Nespu
    Nov 24 '18 at 2:11











  • Hello @RobbiNespu, thanks for checking out the permissions. I have added the public key to gitlab's site. (it isn't self hosted.) Also thanks for the "-vvvv" tip! I'll review that and post it if your answer below doesn't fix my issues

    – Justin Schwimmer
    Nov 26 '18 at 16:57











  • Just to make sure, can you run restorecon -Rv ~/.ssh and test ssh?

    – Robbi Nespu
    Nov 28 '18 at 2:27











  • Running that command produced no results - no files are listed as being changed, it just gave me a command prompt below. Does it matter that my SELinux is disabled?

    – Justin Schwimmer
    Nov 28 '18 at 6:19














0












0








0








My ssh connection to gitlab fails on my fedora 28 box, with no password request needed. This is preventing me from successfully running needed git commands. I've researched this thoroughly within SO and Googs, but nothing I found seems to work for my issue, more importantly, help explain the various sections of a 'vvv' log.



Set Up




Selinux = disabled



SSH directory: /home/justin/.ssh



    drwx------   2 justin   root  4096 Nov 19 22:16  .ssh


Step 1) Create the keys: ssh-keygen -t rsa -b 4096 -C "j.r.schwimmer@domain.com"



I accept the default location and then enter a pass phrase creating:



    -rw------- 1 justin root 3326 Nov 19 22:16 id_rsa

-rw-r--r-- 1 justin root 749 Nov 19 22:16 id_rsa.pub


Step 2) Grab the contents of id_rsa.pub and add the key to Gitlabs SSH section, verifying that the entire public key is entered into Gitlabs form.



Step 3) Verify that the ssh-agent is running: eval "$(ssh-agent -s)"



Step 4) Add the private key: ssh-add /home/justin/.ssh/id_rsa



Step 5) Run my ssh command: ssh -vvvT justin@gitlab.domainname.net
Confirming that the connection should create a known hosts file



Step 6) Cry after seeing: justin@gitlab.domainname.net: Permission denied (publickey)




The Verbose (vvv) Log



Loads config settings including those found in gitlab.conf:



OpenSSH_7.7p1, OpenSSL 1.1.0h-fips  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 2: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/gitlab.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/gitlab.conf


Makes connection:



debug2: resolving "gitlab.domainname.net" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to gitlab.domainname.net [17.17.17.17] port 22.
debug1: Connection established.


Identity File Section...(Not sure what its doing?)



debug1: identity file /home/justin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/justin/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/justin/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
...checking others
debug1: identity file /home/justin/.ssh/id_xmss-cert type -1


More logs that appear to be handshake related (I think?) / known_hosts setup:



debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.1
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.domainname.net:22 as 'justin'
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31


Known host check:



debug1: Server host key: ecdsa-sha2-nistp256 SHA256:hyO5F3tsdf34PQGAYYhmoGW5J/leBnC0hj3IoE1F68
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 17.17.17.17
debug1: Host 'gitlab.domainname.net' is known and matches the ECDSA host key.
debug1: Found key in /home/justin/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0


Final Auth Verfication:



debug1: rekey after 4294967296 blocks
debug2: key: /home/justin/.ssh/id_rsa (0x55d580f2caf0), agent
debug2: key: /home/justin/.ssh/id_dsa ((nil))
debug2: key: /home/justin/.ssh/id_ecdsa ((nil))
debug2: key: /home/justin/.ssh/id_ed25519 ((nil))
debug2: key: /home/justin/.ssh/id_xmss ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51


(A) Is the auth section below responsible for the overall failure?
(B) It seems to be trying to load every key type after successfully trying id_rsa, is that correct?



debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/justin/.ssh/id_dsa
debug3: no such identity: /home/justin/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ecdsa
debug3: no such identity: /home/justin/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ed25519
debug3: no such identity: /home/justin/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_xmss
debug3: no such identity: /home/justin/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
justin@gitlab.domainname.net: Permission denied (publickey).


Failed Tests




  • I've tried setting this all up as a root user... failed

  • Changing key files ownership, groups and permissions... failed

  • I've tried different key types (id_ecdsa)... failed


  • I tried adding a config file to /etc/ssh/ssh_config.d/ that contained the following:



    Host gitlab.com
    StrictHostKeyChecking no
    LogLevel VERBOSE



More Questions



(C) debug1: identity file /home/justin/.ssh/id_rsa type 0: Line within the "Identity File Section", is this 0 value supposed to be 2? What does this value even mean, the number of key files it found per type?



(D) I'm unable to find an auth.log (or any logging for ssh) file within: /var/log/, why would that be or how can I fix this (Fedora 28)?



Thanks you for your time!



UPDATED: The Verbose (vvvv) Log



OpenSSH_7.7p1, OpenSSL 1.1.0h-fips  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 2: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/gitlab.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/gitlab.conf
debug1: /etc/ssh/ssh_config.d/gitlab.conf line 2: Applying options for gitlab.domainname.net
debug2: resolving "gitlab.domainname.net" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to gitlab.domainname.net [17.17.17.17] port 22.
debug1: Connection established.
debug1: identity file /home/justin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/justin/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.1
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.domainname.net:22 as 'justin'
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:hyO5F3t2OPc2nPQGAYYhmoGW5J/leBnC0hj3IoE1F68
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 17.17.17.17
debug1: Host 'gitlab.domainname.net' is known and matches the ECDSA host key.
debug1: Found key in /home/justin/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: /home/justin/.ssh/id_rsa (0x55568d375a80), agent
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
justin@gitlab.domainname.net: Permission denied (publickey).









share|improve this question
















My ssh connection to gitlab fails on my fedora 28 box, with no password request needed. This is preventing me from successfully running needed git commands. I've researched this thoroughly within SO and Googs, but nothing I found seems to work for my issue, more importantly, help explain the various sections of a 'vvv' log.



Set Up




Selinux = disabled



SSH directory: /home/justin/.ssh



    drwx------   2 justin   root  4096 Nov 19 22:16  .ssh


Step 1) Create the keys: ssh-keygen -t rsa -b 4096 -C "j.r.schwimmer@domain.com"



I accept the default location and then enter a pass phrase creating:



    -rw------- 1 justin root 3326 Nov 19 22:16 id_rsa

-rw-r--r-- 1 justin root 749 Nov 19 22:16 id_rsa.pub


Step 2) Grab the contents of id_rsa.pub and add the key to Gitlabs SSH section, verifying that the entire public key is entered into Gitlabs form.



Step 3) Verify that the ssh-agent is running: eval "$(ssh-agent -s)"



Step 4) Add the private key: ssh-add /home/justin/.ssh/id_rsa



Step 5) Run my ssh command: ssh -vvvT justin@gitlab.domainname.net
Confirming that the connection should create a known hosts file



Step 6) Cry after seeing: justin@gitlab.domainname.net: Permission denied (publickey)




The Verbose (vvv) Log



Loads config settings including those found in gitlab.conf:



OpenSSH_7.7p1, OpenSSL 1.1.0h-fips  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 2: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/gitlab.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/gitlab.conf


Makes connection:



debug2: resolving "gitlab.domainname.net" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to gitlab.domainname.net [17.17.17.17] port 22.
debug1: Connection established.


Identity File Section...(Not sure what its doing?)



debug1: identity file /home/justin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/justin/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/justin/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
...checking others
debug1: identity file /home/justin/.ssh/id_xmss-cert type -1


More logs that appear to be handshake related (I think?) / known_hosts setup:



debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.1
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.domainname.net:22 as 'justin'
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31


Known host check:



debug1: Server host key: ecdsa-sha2-nistp256 SHA256:hyO5F3tsdf34PQGAYYhmoGW5J/leBnC0hj3IoE1F68
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 17.17.17.17
debug1: Host 'gitlab.domainname.net' is known and matches the ECDSA host key.
debug1: Found key in /home/justin/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0


Final Auth Verfication:



debug1: rekey after 4294967296 blocks
debug2: key: /home/justin/.ssh/id_rsa (0x55d580f2caf0), agent
debug2: key: /home/justin/.ssh/id_dsa ((nil))
debug2: key: /home/justin/.ssh/id_ecdsa ((nil))
debug2: key: /home/justin/.ssh/id_ed25519 ((nil))
debug2: key: /home/justin/.ssh/id_xmss ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51


(A) Is the auth section below responsible for the overall failure?
(B) It seems to be trying to load every key type after successfully trying id_rsa, is that correct?



debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/justin/.ssh/id_dsa
debug3: no such identity: /home/justin/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ecdsa
debug3: no such identity: /home/justin/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ed25519
debug3: no such identity: /home/justin/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_xmss
debug3: no such identity: /home/justin/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
justin@gitlab.domainname.net: Permission denied (publickey).


Failed Tests




  • I've tried setting this all up as a root user... failed

  • Changing key files ownership, groups and permissions... failed

  • I've tried different key types (id_ecdsa)... failed


  • I tried adding a config file to /etc/ssh/ssh_config.d/ that contained the following:



    Host gitlab.com
    StrictHostKeyChecking no
    LogLevel VERBOSE



More Questions



(C) debug1: identity file /home/justin/.ssh/id_rsa type 0: Line within the "Identity File Section", is this 0 value supposed to be 2? What does this value even mean, the number of key files it found per type?



(D) I'm unable to find an auth.log (or any logging for ssh) file within: /var/log/, why would that be or how can I fix this (Fedora 28)?



Thanks you for your time!



UPDATED: The Verbose (vvvv) Log



OpenSSH_7.7p1, OpenSSL 1.1.0h-fips  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 2: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/gitlab.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/gitlab.conf
debug1: /etc/ssh/ssh_config.d/gitlab.conf line 2: Applying options for gitlab.domainname.net
debug2: resolving "gitlab.domainname.net" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to gitlab.domainname.net [17.17.17.17] port 22.
debug1: Connection established.
debug1: identity file /home/justin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/justin/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.1
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.domainname.net:22 as 'justin'
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:hyO5F3t2OPc2nPQGAYYhmoGW5J/leBnC0hj3IoE1F68
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.domainname.net
debug3: hostkeys_foreach: reading file "/home/justin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/justin/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 17.17.17.17
debug1: Host 'gitlab.domainname.net' is known and matches the ECDSA host key.
debug1: Found key in /home/justin/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: /home/justin/.ssh/id_rsa (0x55568d375a80), agent
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
justin@gitlab.domainname.net: Permission denied (publickey).






logging ssh permissions gitlab fedora






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 26 '18 at 17:47







Justin Schwimmer

















asked Nov 20 '18 at 8:24









Justin SchwimmerJustin Schwimmer

989




989








  • 1





    You ssh folder and key permission are correct and good. Nothing wrong about that. have you add ssh keys to gitlab.com/profile/keys ? if you self hosted gitlab, find it on setting menu

    – Robbi Nespu
    Nov 24 '18 at 1:20








  • 1





    post full ssh -vvvv git@gitlab.com log. This will helping to identify the problem.

    – Robbi Nespu
    Nov 24 '18 at 2:11











  • Hello @RobbiNespu, thanks for checking out the permissions. I have added the public key to gitlab's site. (it isn't self hosted.) Also thanks for the "-vvvv" tip! I'll review that and post it if your answer below doesn't fix my issues

    – Justin Schwimmer
    Nov 26 '18 at 16:57











  • Just to make sure, can you run restorecon -Rv ~/.ssh and test ssh?

    – Robbi Nespu
    Nov 28 '18 at 2:27











  • Running that command produced no results - no files are listed as being changed, it just gave me a command prompt below. Does it matter that my SELinux is disabled?

    – Justin Schwimmer
    Nov 28 '18 at 6:19














  • 1





    You ssh folder and key permission are correct and good. Nothing wrong about that. have you add ssh keys to gitlab.com/profile/keys ? if you self hosted gitlab, find it on setting menu

    – Robbi Nespu
    Nov 24 '18 at 1:20








  • 1





    post full ssh -vvvv git@gitlab.com log. This will helping to identify the problem.

    – Robbi Nespu
    Nov 24 '18 at 2:11











  • Hello @RobbiNespu, thanks for checking out the permissions. I have added the public key to gitlab's site. (it isn't self hosted.) Also thanks for the "-vvvv" tip! I'll review that and post it if your answer below doesn't fix my issues

    – Justin Schwimmer
    Nov 26 '18 at 16:57











  • Just to make sure, can you run restorecon -Rv ~/.ssh and test ssh?

    – Robbi Nespu
    Nov 28 '18 at 2:27











  • Running that command produced no results - no files are listed as being changed, it just gave me a command prompt below. Does it matter that my SELinux is disabled?

    – Justin Schwimmer
    Nov 28 '18 at 6:19








1




1





You ssh folder and key permission are correct and good. Nothing wrong about that. have you add ssh keys to gitlab.com/profile/keys ? if you self hosted gitlab, find it on setting menu

– Robbi Nespu
Nov 24 '18 at 1:20







You ssh folder and key permission are correct and good. Nothing wrong about that. have you add ssh keys to gitlab.com/profile/keys ? if you self hosted gitlab, find it on setting menu

– Robbi Nespu
Nov 24 '18 at 1:20






1




1





post full ssh -vvvv git@gitlab.com log. This will helping to identify the problem.

– Robbi Nespu
Nov 24 '18 at 2:11





post full ssh -vvvv git@gitlab.com log. This will helping to identify the problem.

– Robbi Nespu
Nov 24 '18 at 2:11













Hello @RobbiNespu, thanks for checking out the permissions. I have added the public key to gitlab's site. (it isn't self hosted.) Also thanks for the "-vvvv" tip! I'll review that and post it if your answer below doesn't fix my issues

– Justin Schwimmer
Nov 26 '18 at 16:57





Hello @RobbiNespu, thanks for checking out the permissions. I have added the public key to gitlab's site. (it isn't self hosted.) Also thanks for the "-vvvv" tip! I'll review that and post it if your answer below doesn't fix my issues

– Justin Schwimmer
Nov 26 '18 at 16:57













Just to make sure, can you run restorecon -Rv ~/.ssh and test ssh?

– Robbi Nespu
Nov 28 '18 at 2:27





Just to make sure, can you run restorecon -Rv ~/.ssh and test ssh?

– Robbi Nespu
Nov 28 '18 at 2:27













Running that command produced no results - no files are listed as being changed, it just gave me a command prompt below. Does it matter that my SELinux is disabled?

– Justin Schwimmer
Nov 28 '18 at 6:19





Running that command produced no results - no files are listed as being changed, it just gave me a command prompt below. Does it matter that my SELinux is disabled?

– Justin Schwimmer
Nov 28 '18 at 6:19












1 Answer
1






active

oldest

votes


















1














your ssh folder and ssh key look good, just run ssh-add -l to see your identity, it nothing then run ssh-add it will configure keys itself since it installed on default .ssh directory and identify your identity



Use git username to ssh instead of your username on Gitlab instance. Change your config file into this



    User git
Host gitlab.com gitlab.domainname.net
IdentityFile ~/.ssh/id_rsa
TCPKeepAlive yes
IdentitiesOnly yes
LogLevel VERBOSE


add your ssh key to https://gitlab.com/profile/keys and if you are self hosted find it on setting profile menu.



try to run ssh -vvvv git@gitlab.com to see if it picks up the SSH key. (Do not use sudo)



That is proper step as show above. Below is your QA



A: You RSA key is are not succesfully read



debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51


As I can see ssh client is not able to find separate public key
debug1: identity file /home/justin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory



B: but the RSA key does't work (last line). so it try to load other keys type but not are not exist, anyway we just need one working key.



debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/justin/.ssh/id_dsa
debug3: no such identity: /home/justin/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ecdsa
debug3: no such identity: /home/justin/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ed25519
debug3: no such identity: /home/justin/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_xmss
debug3: no such identity: /home/justin/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.


Re-check your SELINUX status. Grep "denied" keyword from audit log. It no need to disable / change SELINUX, if blocked try to fix with restorecon -Rv ~/.ssh






share|improve this answer


























  • Thanks again for your response! Let's see: ssh-add -l shows that my key has been loaded. Using either my gitlab or git accounts doesn't change the result Changing my git config file to the recommended one produced a "garbage at end of line" error until I modified "Hostname" to "Host"

    – Justin Schwimmer
    Nov 26 '18 at 17:49











  • Your answer to the questions make sense, (A) the line in question just threw me off because it changed from '-1' to zero. I've modified my OG question to include the "-vvvv" log. The main thing I'm confused about from this log is that toward the end it appears like my public key was loaded: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa

    – Justin Schwimmer
    Nov 26 '18 at 17:49











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53388875%2ffedora-gitlab-ssh-denied-help-understanding-verbose-logs%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









1














your ssh folder and ssh key look good, just run ssh-add -l to see your identity, it nothing then run ssh-add it will configure keys itself since it installed on default .ssh directory and identify your identity



Use git username to ssh instead of your username on Gitlab instance. Change your config file into this



    User git
Host gitlab.com gitlab.domainname.net
IdentityFile ~/.ssh/id_rsa
TCPKeepAlive yes
IdentitiesOnly yes
LogLevel VERBOSE


add your ssh key to https://gitlab.com/profile/keys and if you are self hosted find it on setting profile menu.



try to run ssh -vvvv git@gitlab.com to see if it picks up the SSH key. (Do not use sudo)



That is proper step as show above. Below is your QA



A: You RSA key is are not succesfully read



debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51


As I can see ssh client is not able to find separate public key
debug1: identity file /home/justin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory



B: but the RSA key does't work (last line). so it try to load other keys type but not are not exist, anyway we just need one working key.



debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/justin/.ssh/id_dsa
debug3: no such identity: /home/justin/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ecdsa
debug3: no such identity: /home/justin/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ed25519
debug3: no such identity: /home/justin/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_xmss
debug3: no such identity: /home/justin/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.


Re-check your SELINUX status. Grep "denied" keyword from audit log. It no need to disable / change SELINUX, if blocked try to fix with restorecon -Rv ~/.ssh






share|improve this answer


























  • Thanks again for your response! Let's see: ssh-add -l shows that my key has been loaded. Using either my gitlab or git accounts doesn't change the result Changing my git config file to the recommended one produced a "garbage at end of line" error until I modified "Hostname" to "Host"

    – Justin Schwimmer
    Nov 26 '18 at 17:49











  • Your answer to the questions make sense, (A) the line in question just threw me off because it changed from '-1' to zero. I've modified my OG question to include the "-vvvv" log. The main thing I'm confused about from this log is that toward the end it appears like my public key was loaded: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa

    – Justin Schwimmer
    Nov 26 '18 at 17:49
















1














your ssh folder and ssh key look good, just run ssh-add -l to see your identity, it nothing then run ssh-add it will configure keys itself since it installed on default .ssh directory and identify your identity



Use git username to ssh instead of your username on Gitlab instance. Change your config file into this



    User git
Host gitlab.com gitlab.domainname.net
IdentityFile ~/.ssh/id_rsa
TCPKeepAlive yes
IdentitiesOnly yes
LogLevel VERBOSE


add your ssh key to https://gitlab.com/profile/keys and if you are self hosted find it on setting profile menu.



try to run ssh -vvvv git@gitlab.com to see if it picks up the SSH key. (Do not use sudo)



That is proper step as show above. Below is your QA



A: You RSA key is are not succesfully read



debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51


As I can see ssh client is not able to find separate public key
debug1: identity file /home/justin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory



B: but the RSA key does't work (last line). so it try to load other keys type but not are not exist, anyway we just need one working key.



debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/justin/.ssh/id_dsa
debug3: no such identity: /home/justin/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ecdsa
debug3: no such identity: /home/justin/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ed25519
debug3: no such identity: /home/justin/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_xmss
debug3: no such identity: /home/justin/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.


Re-check your SELINUX status. Grep "denied" keyword from audit log. It no need to disable / change SELINUX, if blocked try to fix with restorecon -Rv ~/.ssh






share|improve this answer


























  • Thanks again for your response! Let's see: ssh-add -l shows that my key has been loaded. Using either my gitlab or git accounts doesn't change the result Changing my git config file to the recommended one produced a "garbage at end of line" error until I modified "Hostname" to "Host"

    – Justin Schwimmer
    Nov 26 '18 at 17:49











  • Your answer to the questions make sense, (A) the line in question just threw me off because it changed from '-1' to zero. I've modified my OG question to include the "-vvvv" log. The main thing I'm confused about from this log is that toward the end it appears like my public key was loaded: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa

    – Justin Schwimmer
    Nov 26 '18 at 17:49














1












1








1







your ssh folder and ssh key look good, just run ssh-add -l to see your identity, it nothing then run ssh-add it will configure keys itself since it installed on default .ssh directory and identify your identity



Use git username to ssh instead of your username on Gitlab instance. Change your config file into this



    User git
Host gitlab.com gitlab.domainname.net
IdentityFile ~/.ssh/id_rsa
TCPKeepAlive yes
IdentitiesOnly yes
LogLevel VERBOSE


add your ssh key to https://gitlab.com/profile/keys and if you are self hosted find it on setting profile menu.



try to run ssh -vvvv git@gitlab.com to see if it picks up the SSH key. (Do not use sudo)



That is proper step as show above. Below is your QA



A: You RSA key is are not succesfully read



debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51


As I can see ssh client is not able to find separate public key
debug1: identity file /home/justin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory



B: but the RSA key does't work (last line). so it try to load other keys type but not are not exist, anyway we just need one working key.



debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/justin/.ssh/id_dsa
debug3: no such identity: /home/justin/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ecdsa
debug3: no such identity: /home/justin/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ed25519
debug3: no such identity: /home/justin/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_xmss
debug3: no such identity: /home/justin/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.


Re-check your SELINUX status. Grep "denied" keyword from audit log. It no need to disable / change SELINUX, if blocked try to fix with restorecon -Rv ~/.ssh






share|improve this answer















your ssh folder and ssh key look good, just run ssh-add -l to see your identity, it nothing then run ssh-add it will configure keys itself since it installed on default .ssh directory and identify your identity



Use git username to ssh instead of your username on Gitlab instance. Change your config file into this



    User git
Host gitlab.com gitlab.domainname.net
IdentityFile ~/.ssh/id_rsa
TCPKeepAlive yes
IdentitiesOnly yes
LogLevel VERBOSE


add your ssh key to https://gitlab.com/profile/keys and if you are self hosted find it on setting profile menu.



try to run ssh -vvvv git@gitlab.com to see if it picks up the SSH key. (Do not use sudo)



That is proper step as show above. Below is your QA



A: You RSA key is are not succesfully read



debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51


As I can see ssh client is not able to find separate public key
debug1: identity file /home/justin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory



B: but the RSA key does't work (last line). so it try to load other keys type but not are not exist, anyway we just need one working key.



debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/justin/.ssh/id_dsa
debug3: no such identity: /home/justin/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ecdsa
debug3: no such identity: /home/justin/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_ed25519
debug3: no such identity: /home/justin/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/justin/.ssh/id_xmss
debug3: no such identity: /home/justin/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.


Re-check your SELINUX status. Grep "denied" keyword from audit log. It no need to disable / change SELINUX, if blocked try to fix with restorecon -Rv ~/.ssh







share|improve this answer














share|improve this answer



share|improve this answer








edited Nov 28 '18 at 2:35

























answered Nov 24 '18 at 1:50









Robbi NespuRobbi Nespu

140112




140112













  • Thanks again for your response! Let's see: ssh-add -l shows that my key has been loaded. Using either my gitlab or git accounts doesn't change the result Changing my git config file to the recommended one produced a "garbage at end of line" error until I modified "Hostname" to "Host"

    – Justin Schwimmer
    Nov 26 '18 at 17:49











  • Your answer to the questions make sense, (A) the line in question just threw me off because it changed from '-1' to zero. I've modified my OG question to include the "-vvvv" log. The main thing I'm confused about from this log is that toward the end it appears like my public key was loaded: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa

    – Justin Schwimmer
    Nov 26 '18 at 17:49



















  • Thanks again for your response! Let's see: ssh-add -l shows that my key has been loaded. Using either my gitlab or git accounts doesn't change the result Changing my git config file to the recommended one produced a "garbage at end of line" error until I modified "Hostname" to "Host"

    – Justin Schwimmer
    Nov 26 '18 at 17:49











  • Your answer to the questions make sense, (A) the line in question just threw me off because it changed from '-1' to zero. I've modified my OG question to include the "-vvvv" log. The main thing I'm confused about from this log is that toward the end it appears like my public key was loaded: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa

    – Justin Schwimmer
    Nov 26 '18 at 17:49

















Thanks again for your response! Let's see: ssh-add -l shows that my key has been loaded. Using either my gitlab or git accounts doesn't change the result Changing my git config file to the recommended one produced a "garbage at end of line" error until I modified "Hostname" to "Host"

– Justin Schwimmer
Nov 26 '18 at 17:49





Thanks again for your response! Let's see: ssh-add -l shows that my key has been loaded. Using either my gitlab or git accounts doesn't change the result Changing my git config file to the recommended one produced a "garbage at end of line" error until I modified "Hostname" to "Host"

– Justin Schwimmer
Nov 26 '18 at 17:49













Your answer to the questions make sense, (A) the line in question just threw me off because it changed from '-1' to zero. I've modified my OG question to include the "-vvvv" log. The main thing I'm confused about from this log is that toward the end it appears like my public key was loaded: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa

– Justin Schwimmer
Nov 26 '18 at 17:49





Your answer to the questions make sense, (A) the line in question just threw me off because it changed from '-1' to zero. I've modified my OG question to include the "-vvvv" log. The main thing I'm confused about from this log is that toward the end it appears like my public key was loaded: Offering public key: RSA SHA256:xbGUrKDgYhF34QYM1s20flfoVcjegZpbMwDDbpTmF1c /home/justin/.ssh/id_rsa

– Justin Schwimmer
Nov 26 '18 at 17:49


















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53388875%2ffedora-gitlab-ssh-denied-help-understanding-verbose-logs%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

MongoDB - Not Authorized To Execute Command

How to fix TextFormField cause rebuild widget in Flutter

Npm cannot find a required file even through it is in the searched directory