Unable to connect to Plesk mail server via an email client: TLS handshaking: SSL_accept() failed

Symptoms

• When connecting to a mail server over SSL via an email client (e.g. Outlook), connection fails:

  Your server does not support the connection encryption type you have specified. Try changing the encryption method.

• One of the following error messages appears in the /var/log/maillog logfile:

  postfix/smtpd[25460]: warning: TLS library problem: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol:s23_srvr.c:640:

  dovecot: pop3-login: Disconnected (no auth attempts in 0 secs): user=<>, rip=203.0.113.2, lip=203.0.113.3, TLS handshaking: SSL_accept() failed: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

  Apr 4 15:09:46 server dovecot: imap-login: Disconnected: Connection closed: SSL_accept() failed: error:0A000102:SSL routines::unsupported protocol (no auth attempts in 0 secs): user=<>, rip=203.0.113.2, lip=203.0.113.3, TLS handshaking: SSL_accept() failed: error:0A000102:SSL routines::unsupported protocol, session=<ek3VukYVN8hervMa>

Cause

TLSv1, TLS 1.1, SSLv2 and SSLv3 protocol support, which is required for old email clients old software (e.g, WinHTTP-based applications on Windows 7) is disabled in the Postfix and Dovecot configuration intentionally, because all of these releases have known security issues caused by vulnerabilities and are therefore not recommended for use.

The full history of SSL and TLS releases is as follows:

1995 - SSL 1.0 – never released publicly due to known security issues
1995 - SSL 2.0 is released – deprecated in 2011 due to known security issues
1996 - SSL 3.0 is released – deprecated in 2015 due to known security issues
1999 - TLS 1.0 is released as an upgrade to SSL 3.0 - deprecated in 2020 due to known security issues
2006 - TLS 1.1 is released - deprecated in 2020 due to known security issues
2008 - TLS 1.2 is released - it can still be used, but is considered safe only when weak ciphers and algorithms are removed
2018 - TLS 1.3 is released - recommended, it comes with no known vulnerabilities, improves performance and will not be deprecated anytime soon

Resolution

The recommended solution is to use the latest available version of email clients that support connections over TLS 1.2 and TLS 1.3 or just switch to a mail client or mail-related software that Support TLS 1.2 and TLS 1.3.

For WinHTTP-based applications, you may refer to the following official Microsoft article.

If for some reason changing the software that is used during the email connection attempt is not possible, you may apply what is necessary via either of the following workarounds.

Warning: All of the below configurations are not recommended due to security vulnerabilities tied to the older TLS and SSL protocols. Apply them only at your own risk!

Note: This guide is intended for Plesk administrators. If you are a domain owner, please contact your service provider for assistance with email account configuration.