www.rodneybeede.com "I would love to change the world, but they won't give me the source code" - unknown
 Navigation

Apache Jakarta HTTPClient, Slow Downloads of Large Files - Last Modified 2008-11-25 22:10 UTC - Created 2008-11-25 22:10 UTC

So I was writing a Java application that used HTTPClient 3.1. For some reason downloading a 32MiB file from my local web server on the same machine was taken at least 16 seconds to complete whereas in a web browser it took less than a second.

After many hours of writing Java socket test code and trying different web servers I discovered the issue. It wasn't related to any of my buffer sizes (I tried all kinds of buffers and sizes).

Turns out that because I didn't initialize log4j (I didn't use it) that the logging mechanism on Windows for HTTPClient was wasting lots of time with internal errors from the library logging code. Turning on trace logging actually made things go faster.

A quick fix since I didn't care about the logging for HTTPClient was to do

System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");

This disables all logging. You should set this up before creating any HTTPClient instances.