MATLAB: How to configure MATLAB to allow access to self-signed HTTPS servers

MATLABweb

I am trying to access an HTTPS server with a self-signed certificate from MATLAB. I have tried various methods in various MATLAB releases:

>> urlread('https://self-signed.badssl.com/')

In all MATLAB releases throws:

Error using urlreadwrite (line 98)

Error downloading URL. Your network connection may be down or your proxy settings improperly configured.

Error in urlread (line 36)

[s,status] = urlreadwrite(mfilename,catchErrors,url,varargin{:});

And when trying to use pure Java:

>> u = java.net.URL('https://self-signed.badssl.com/');
>> conn = u.openConnection;
>> conn.connect

I receive:

Java exception occurred:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to

find valid certification path to requested target

at sun.security.ssl.Alerts.getSSLException(Unknown Source)

at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)

at sun.security.ssl.Handshaker.fatalSE(Unknown Source)

(snip)

Further, webread:

>> webread('https://self-signed.badssl.com/')

In MATLAB releases prior to R2016b actually returns the data without any error or warning for this server:

ans =
<!DOCTYPE html>
<html>
<head>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="shortcut icon" href="/icons/favicon-red.ico"/>
  <link rel="apple-touch-icon" href="/icons/icon-red.png"/>
  <title>self-signed.badssl.com</title>
  <link rel="stylesheet" href="/style.css">
  <style>body { background: red; }</style>
</head>
<body>
<div id="content">
  <h1 style="font-size: 12vw;">
    self-signed.<br>badssl.com
  </h1>
</div>
</body>
</html>

But for another server:

>> webread('https://localhost/')

I receive:

Error using readContentFromWebService (line 45)

The server returned the message: "sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find

valid certification path to requested target" for URL, 'https://localhost/' (with HTTP response code unknown).

Error in webread (line 122)

[varargout{1:nargout}] = readContentFromWebService(connection, options);

And in MATLAB release R2016b:

>> webread('https://self-signed.badssl.com/')

Throws:

Error using webread (line 119)

Could not establish a secure connection to "self-signed.badssl.com". The reason is "error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed".

Check your certificate file (C:\MATLAB\R2016b\sys\certificates\ca\rootcerts.pem) for expired, missing or invalid certificates.

For both servers actually.

Best Answer

Before continuing please note that certificates are used for a reason and an untrusted certificate may indicate that communication with the website that you are trying to access may not be secure; you may even be accessing a different website than you might have expected (also see the further explanations in your webbrowser when trying to access the website).

MATLAB verifies HTTPS server certificates in a number of different ways.

WHEN YOU ARE WORKING WITH URLREAD/URLWRITE OR JAVA CLASSES DIRECTLY, the verification is basically performed by the JRE in MATLAB. The JRE uses a keystore with trusted certificate authorities to determine which certificates are trusted. I.e. it only accepts certificates which have been signed by a trusted authority and self-signed certificates are not accepted. If you would like to add a (self-signed) certificate or authority to this store, use the following steps:

1. Download the certificate using your web browser/operating system as CER-or CRT-file.

2. Use the attached MATLAB function to add this certificate as trusted to MATLAB's JRE's keystore. You will need to manually type "yes" when prompted to actually accept the certificate.

3. Restart MATLAB after importing the certificate into the keystore.

Note that the steps above require read/write permissions for the following file and the directory in which it is located:

fullfile(matlabroot,'sys','java','jre',computer('arch'),'jre','lib','security','cacerts')

This means that you may need to start MATLAB as Administrator/root depending on where MATLAB is installed and the permissions set on this location.

WHEN WORKING WITH WEBREAD/WEBWRITE/WEBSAVE, there are two major situations:

1. You are connecting to a server with basic or no further authentication whatsoever. In this case webread/webwrite/websave only performs its own verification. But again there two situations:

a. In releases prior to R2016b the verification is limited to only verifying that the URL you are accessing matches the CN in the certificate; there is no validation of the certificate's authenticity however. I.e. it accepts self-signed certificates as long as they are valid for the server in question.

b. In release R2016b it first verifies the authenticity of the certificates using a keystore similar to- but separate from- the keystore of the JRE mentioned above. I.e. it only accepts certificates which have been signed by a trusted authority. If this validation fails, you receive the error: ERROR: Error using webread (line 119) Could not establish a secure connection to "self-signed.badssl.com". The reason is "error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed". Check your certificate file (C:\MATLAB\R2016b\sys\certificates\ca\rootcerts.pem) for expired, missing or invalid certificates.

If you want to access a server with a self-signed certificate anyway you can either:

i) Add the certificate to the trusted keystore of webread/webwrite/webstore, see the documentation of weboptions to learn more about this:

http://www.mathworks.com/help/releases/R2016b/matlab/ref/weboptions.html#expand_panel_body_input_argument_namevalue_CertificateFilename

ii) Disable the authenticity verification (i.e. fall back to the behavior in older releases), as noted on the documentation page mentioned above this can be accomplished by setting 'CertificateFilename' to empty:

>> o = weboptions('CertificateFilename','');
>> webread('https://self-signed.badssl.com/',o)

2. You are connecting to a server with authentication other than basic (e.g. NTLM). In this case webread/webwrite/websave first performs the verification described under point 1 but then falls back to using a Java interface which actually also performs its own authentication. If this part fails, you receive the error: ERROR: Error using readContentFromWebService (line 45) The server returned the message: "sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target" for URL, 'https://localhost/' (with HTTP response code unknown). Error in webread (line 122) [varargout{1:nargout}] = readContentFromWebService(connection, options);

See the "WHEN YOU ARE WORKING WITH URLREAD/URLWRITE OR JAVA CLASSES DIRECTLY" section above on how to make this Java part trust your self-signed servers.

Note that this means that if you want to work with webread, a self-signed HTTPS server with NTLM authentication in MATLAB R2016b, you need to actually add the self-signed certificate in two places: the keystore specified by CertificateFilename (or set this to empty) and the JRE keystore.

Note: Above script did not work for older MATLAB installs, such as R2007b, as the directory structure is different for the JRE that comes with the older MATLAB. Pasting something similar to below into a command prompt, allowed the certificate to be added. However there were new java issues afterward, such as a missing server hello.

"C:\Program Files\MATLAB\R2007b\sys\java\jre\win64\jre1.6.0\bin\keytool.exe" -import -file C:\temp\My_Downloaded_Cert_from_Chrome.cer -keystore "C:\Program Files\MATLAB\R2007b\sys\java\jre\win64\jre1.6.0\lib\security\cacerts" -storepass changeit

Related Solutions

MATLAB: Can’t I reconnect to the BLE device using its name after its MAC address changes

The reason that multiple devices with the same name were found is that MATLAB caches all previously found devices in the system to avoid forcing users to scan for the device for each MATLAB session. To work around this issue, use the blelist command to scan for the desired device and identify the new address of the device. With this address, connect to it using the following command:

>> b = ble(address);

For more information regarding connecting to a BLE device, refer to the following documentation page:

https://www.mathworks.com/help/matlab/import_export/find-your-bluetooth-low-energy-peripheral-devices.html

MATLAB: How to preemptively include a Basic Authentication header when working with “webread”/”webwrite”/”websave” in MATLAB R2019b

In MATLAB releases prior to MATLAB R2019b, when you specified a 'Username' and 'Password' MATLAB would preemptively pass along these in a Basic Authentication header in the first request which "webread" makes. In MATLAB R2019b this changed, MATLAB now follows the challenge and response workflows from RFC-7235 more closely; which basically means it first makes a request without credentials expecting the server to return an "401 - Unauthorized" response in combination with a 'WWW-Authenticate' header specifying which authentication methods it supports. MATLAB then makes a second request with that particular authentication method. This is a more secure approach but this breaks workflows working with servers like the one described above.

If you still want "webread"/"webwrite"/"websave" to preemptive include a Basic Authentication header in the first request, do not specify 'Username' and 'Password' property/value pairs in "weboptions", instead use 'HeaderFields' to specify your own Authentication header. A Basic Authentication header is very simple to form: it is simply the text 'Basic ', followed by [username] colon [password] in base64 encoded format. So in MATLAB you could for example write:

% Example username and password
username = 'myUser';
password = 'myPassword';
% Manually set Authorization header field in weboptions
options = weboptions('HeaderFields',{'Authorization',...
    ['Basic ' matlab.net.base64encode([username ':' password])]});
% Make a request using these options
webread('https://httpbin.org/hidden-basic-auth/myUser/myPassword', options)

Note httpbin.org is a service for testing HTTP requests, the 'hidden-basic-auth' endpoint behaves in a way similar to described above (the request succeeds if you preemptively include a Basic Authorization header but immediately returns "404 - Not Found" if no (valid) authentication header was present). The myUser and myPassword specified in the request URL here is part of this particular testing endpoint, when calling your actual server you would normally not include your username and password in the request URL in this way, also do not call httpbin.org with you real username and password.

This approach also works correctly in MATLAB releases prior to R2019b.

Further note that the workflow for the HTTP interface has not changed. In this interface, if you explicitly specify 'Username', 'Password' and exactly one authentication method using a Credentials object, the Authorization header was always- and still is- preemptively included in the first request. So you can still write:

import matlab.net.http.*
% Create a credentials object. If you specify exactly one Scheme, we
% immediately automatically add the Authorization header for this Scheme,
% if you do not specify a Scheme we first make a request without any
% Authorization header expecting a server challenge.
creds = Credentials('Username','myUser','Password','myPassword',...
    'Scheme', AuthenticationScheme.Basic);
% Add these credentials to the options
options = HTTPOptions('Credentials', creds);
% Create a request
req = RequestMessage('GET');
% Execute this request on some URL with these options
response = req.send('https://httpbin.org/hidden-basic-auth/myUser/myPassword',options);

Best Answer

Related Solutions

MATLAB: Can’t I reconnect to the BLE device using its name after its MAC address changes

MATLAB: How to preemptively include a Basic Authentication header when working with “webread”/​”webwrite”​/”websave” in MATLAB R2019b

Related Question

MATLAB: How to preemptively include a Basic Authentication header when working with “webread”/”webwrite”/”websave” in MATLAB R2019b