MATLAB: Urlread-function is very slow since R2012b – why

MATLABr2012burlread

Hi,

I'm using a small piece of code to read out information from about 100 websites by using urlread in its easiest form:

for count = 1:elements
        html_str = urlread(['http://www.website.com/page' num2str(count)]);
end

Yesterday I changed my Matlab-version from R2011a to R2012b and was slightly shocked. With the old version the loop took about 20-30 seconds. The new one needs nearly 5 Minutes!

My research quarry that Mathworks has expanded the urlread-function with 3 additional parameters and took some old ones away. http://www.mathworks.de/de/help/matlab/release-notes.html But I do not use any of these parameters, only the naked function. I tried the manual reduction of the 'Timeout' to 0.1, but with no influence.

Are there any ideas, what causes the speed-loss and first of all how I can get my old performance back? Thanks a lot for your assistance!

Best Answer

Why do you assume that the problem is caused by Matlab? Do you have the old Matlab still installed? If so, please check if the old version is slower today also.

Related Solutions

MATLAB: Getting full HTTP response to HTTP request in urlread

Here's my answer to this problem, although it might have some bugs ...

getHeaderField was needed, note getHeaderField(0) is the response like 400 or 401

In addition, I added getErrorStream, which the server I am working with sends when there is an error

I still don't like the stream copier ...

allHeaders = struct('Response',char(urlConnection.getHeaderField(0)));
done = false;
headerIndex = 0;
while ~done
    headerIndex = headerIndex + 1;
    headerValue = char(urlConnection.getHeaderField(headerIndex));
    if ~isempty(headerValue)
        headerName = char(urlConnection.getHeaderFieldKey(headerIndex));
        headerName(headerName == '-') = '_';
        allHeaders(1).(headerName) = headerValue;
    else
        done = true;
    end
end
status = struct('value',urlConnection.getResponseCode(),...
    'msg',char(urlConnection.getResponseMessage));
isGood = true;
try
    inputStream = urlConnection.getInputStream;
catch ME
    try
        inputStream = urlConnection.getErrorStream;
    catch ME %#ok<*NASGU>
        isGood = false;
    end
end
if isGood
    byteArrayOutputStream = java.io.ByteArrayOutputStream;
    % This StreamCopier is unsupported and may change at any time. OH GREAT :/
    isc = InterruptibleStreamCopier.getInterruptibleStreamCopier;
    isc.copyStream(inputStream,byteArrayOutputStream);
    inputStream.close;
    byteArrayOutputStream.close;
    output = native2unicode(typecast(byteArrayOutputStream.toByteArray','uint8'),'UTF-8');
else
    output = '';
end

MATLAB: Matlab using Java openStream(url) get :Java exception occurred: java.lang.NullPointerException

Yes. so the answer is Matlab urlread does not support authentication with lower level request. So we need to code firectly against the Java URL class to use them:

% Matlab's urlread() doesn't do HTTP Request params, so work directly with Java
jUrl = java.net.URL(url);
conn = jUrl.openConnection();
conn.setRequestProperty('Authorization', ['Basic ' base64encode([user ':' password])]);
conn.connect();
info.status = conn.getResponseCode();
info.errMsg = char(readstream(conn.getErrorStream()));
s = readstream(conn.getInputStream());
function out = base64encode(str)
% Uses Sun-specific class, but we know that is the JVM Matlab ships with
encoder = sun.misc.BASE64Encoder();
out = char(encoder.encode(java.lang.String(str).getBytes()));
function out = readstream(inStream)
%READSTREAM Read all bytes from stream to uint8
try
    import com.mathworks.mlwidgets.io.InterruptibleStreamCopier;
    byteStream = java.io.ByteArrayOutputStream();
    isc = InterruptibleStreamCopier.getInterruptibleStreamCopier();
    isc.copyStream(inStream, byteStream);
    inStream.close();
    byteStream.close();
    out = typecast(byteStream.toByteArray', 'uint8'); %'
catch err
    out = []; %HACK: quash
end

Hope it will help others later.

Best Answer

Related Solutions

MATLAB: Getting full HTTP response to HTTP request in urlread

MATLAB: Matlab using Java openStream(url) get :Java exception occurred: java.lang.​NullPointe​rException

Related Question

MATLAB: Matlab using Java openStream(url) get :Java exception occurred: java.lang.NullPointerException