What does the "wrong signature type" SSL error means?

In updown.io or in your own manual tests (using curl, wget, some API clients, etc..) you may encouter one of the following error when trying to query your https website/API:
SSL_connect returned=1 errno=0 state: error wrong signature type 
curl: (35) error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
This error means that the TLS client (OpenSSL in updown.io case) is receiving a signature type which is considered invalid, and this happens more with recent versions of OpenSSL because some signature algorithms have been deprecated in the past years due to vulnerabilities.

The error is not very precise though (there are a lot of different "signatures" in a TLS handshake), for example it could come from one older certificate signed using SHA-1 instead of SHA-256, but this is not very common now (except some cross-signed root for backward compatibility) and it usually doesn't cause any problem because there's another SHA-256 certification path that modern client can use.

For example at the time of writing let's encrypt certificates are still cross-signed with a SHA-1 signed root certificate (DST Root CA X3) for additional client compatibility, but most clients will actually use the other chain with contains only SHA-256 signed certificates and ends on the ISRG Root X1 root certificate.

So from what I investigated so far when this error occurs it looks like this is related to the SHA-1 signing algorithm being used for "Peer signing digest" (which is the algorithm used to sign communications from the server to the client during the handshake). What happens is that even if the client asks for another (more secure) signing algorithm (which modern clients do), the server always use SHA1 anyway. And in recent versions of OpenSSL this is considered invalid because of the deprecation mentioned above. This is probably caused by outdated software which does not support changing algorithm (new feature since TLS 1.2). Here is an example of how it looks on a sample website (no specific example is given here, you can set the HOSTNAME variable to your website's hostname to test it).
> HOSTNAME=www.yourwebsite.com
> curl https://$HOSTNAME
curl: (35) error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
Same error message when testing with openssl directly (this test is using OpenSSL 1.1.1l):
> echo | openssl s_client -servername $HOSTNAME -connect $HOSTNAME:443
[...]
140479360882496:error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type:../ssl/t1_lib.c:1145: [...]
And If we can manage to test this from an older OpenSSL version (less strict about this, OpenSSL 1.1.1 in this case), we can see the peer signature algorithm does not honor what is asked:
> echo | openssl s_client -servername $HOSTNAME -connect $HOSTNAME:443 -sigalgs "RSA-PSS+SHA256" | fgrep "Peer sign"
[...]
Peer signing digest: SHA1
Peer signature type: RSA
Here we try to contact the server asking for a specific signature algorithm (RSA-PSS+SHA256) but no matter what we try to specify the server still uses RSA-SHA1.

If we try the same thing on updown.io for example (which doesn't exibit this problem):
> echo | openssl s_client -servername updown.io -connect updown.io:443 -tls1_2 -sigalgs "RSA-PSS+SHA256" | fgrep "Peer sign"
[...]
Peer signing digest: SHA256
Peer signature type: RSA-PSS
We have the correct algorithm, and if we ask to use something else, it works too:
> echo | openssl s_client -servername updown.io -connect updown.io:443 -tls1_2 -sigalgs "RSA+SHA1" | fgrep "Peer sign"
[...] Peer signing digest: SHA1 Peer signature type: RSA
Or it fails it it's not supported (here using tls1.3 this signature algorithm is not supported):
> echo | openssl s_client -servername updown.io -connect updown.io:443 -tls1_3 -sigalgs "RSA+SHA1" | fgrep "Peer sign"
140696053629376:error:14215076:SSL routines:tls12_copy_sigalgs:no suitable signature algorithm:../ssl/t1_lib.c:1643:
140696053629376:error:141DD044:SSL routines:tls_construct_ctos_sig_algs:internal error:../ssl/statem/extensions_clnt.c:281:
So the problem here in my opinion is an outdated software stack handling TLS on the server side (e.g. OpenSSL, LibreSSL, etc...), maybe the web server (nginx, apache, etc...) also have a play here I am not sure. I would recommending upgrading this TLS software to see if it supports other signing algorithms.

This may not cause any problem in browsers at the moment (because they have a permissive implementation), but it may at some point, and it's already impacting stricter clients like curl or other programming HTTP APIs. That is why updown.io will not try to ignore this error (which is theoretically possible by lowering security settings for OpenSSL).

If you encouter this issue and manage to fix it, please let me know what software version you were using for TLS so I can give more precisions here. Thanks!

Feedback and Knowledge Base