The MATLAB 2020a installer contains its own shared library libcrypto.so.1.1, but uses the system library libk5crypto.so.3. System library libk5crypto.so.3 uses new symbols, which system library libcrypto.so.1.1 has, but MATLAB's library libcrypto.so.1.1 doesn't. So, you have to delete the MATLAB installer's copy of libcrypto.so.1.1. The sequence of commands that worked for me in RHEL 8.2 were:
cd /home/YourUserName/Downloads/matlab_R2020a_glnxa64/bin/glnxa64
rm libcrypto.so.1.1
sudo su -
cd /home/YourUserName/Downloads/matlab_R2020a_glnxa64
export DISPLAY=':0'
./install
That got the installer to work.
P.S. -- When I got to the step of entering my MathWorks account password in the resulting MATLABWindow application, the password field wouldn't let me type until I selected "Show Details" from the "MathWorks Product Installer" drop-down menu in the top menu bar and then clicked back into the main screen of the MATLABWindow application. Basically, I think I had to remove and restore focus on the MATLABWindow application main screen to get the password field working again.
Best Answer