martin-evans.me.uk Always finish the day on a success

What module do you use when doing any kind of HTTP programming - LWP? By far the most popular module for the HTTP protocol in Perl and one which just works and works well with a minimal amount of application code. Now you might be put off by the 123 bug reports in the LWP rt queue but in all my time I've never hit one myself - other than an issue which was not LWP but Perl and to do with tainted data used with LWP (see Why is taint mode so slow?).

I've recently been trying to speed up a small application that makes a huge number of HTTP GET and small POST requests. One important area was that we needed to support cookies as the initial POST is to login (where a cookie is set) and subsequent POSTs need to pass the cookie. Whenever I profile it with Devel::NYTProf (thank you Tim) LWP related modules come up at the top (although in interesting areas like select()). Knowing that LWP is a lot of Perl code the obvious area to look at was an XS module with more compiled C code. We tried GHTTP, Furl, and WWW::Curl::Easy in addition to LWP and there was a clear winner (for us) - WWW::Curl::Easy. Comparing the code using LWP and WWW::Curl::Easy for a simple GET:

and with WWW::Curl::Easy:

So, not much more code but a little extra to write the HTTP headers separately from the body which makes parsing them easier later. You also need to remember that since Curl is writing the header data to a scalar via a file handle you need to seek to the start and truncate it between each request unless you set WRITEHEADER each time. Of course, reading the headers is considerably easier in LWP.

WWW::Curl::Easy is a fairly simple wrapper around libcurl which is written in C. You might initially be put off by the lack of pod but in fact it does not need much as it is just a wrapper around libcurl so most of the documentation is in libcurl. The equivalent LWP and WWW::Curl::Easy code shows quite clearly that LWP is far more concise (in Perl terms) but the speedup we got using libcurl was worth having. On a simple GET request with 2000 GETs we got:

LWP: 9 wallclock secs ( 5.76 usr + 1.05 sys = 6.81 CPU) @ 293.69/s (n=2000)
GHTTP: 3 wallclock secs ( 0.50 usr + 0.70 sys = 1.20 CPU) @ 1666.67/s (n=2000)
Curl: 3 wallclock secs ( 0.26 usr + 0.25 sys = 0.51 CPU) @ 3921.57/s (n=2000)
Furl: 3 wallclock secs ( 0.93 usr + 0.22 sys = 1.15 CPU) @ 1739.13/s (n=2000)

So far so good. Changing to WWW::Curl::Easy seemed a no brainer except when I tried POSTs it was massively slower. Strangely, it was far less CPU time but more than twice as slow in real time:

Benchmark: timing 1000 iterations of curl, lwp...
curl: 56 wallclock secs ( 0.16 usr + 0.09 sys = 0.25 CPU) @ 4000.00/s (n=1000)
(warning: too few iterations for a reliable count)
lwp: 22 wallclock secs ( 4.29 usr + 0.60 sys = 4.89 CPU) @ 204.50/s (n=1000)

Those numbers seemed strange to me; why was Curl apparently waiting because clearly it was spending far less CPU time. A quick experiment with the Curl options showed that setting either CURLOPT_TCP_NODELAY or CURLOPT_FORBID_REUSE changed the results to:

Benchmark: timing 1000 iterations of curl, lwp...
curl: 18 wallclock secs ( 0.28 usr + 0.10 sys = 0.38 CPU) @ 2631.58/s (n=1000)
(warning: too few iterations for a reliable count)
lwp: 22 wallclock secs ( 4.33 usr + 0.61 sys = 4.94 CPU) @ 202.43/s (n=1000)

But the nagle algorithm surely is not involved here as I was only doing one POST and surely Curl was doing only a single write to the socket! I'd looked at the WWW::Curl::Easy pod and it had a nice formadd method where you can add form arguments so my code looked like:

Examining an strace of the code I could see 2 writes to the socket connected to the HTTP server, the first for the HTTP header and the second for the POSTed data (which is very small). This is how nagle intervenes as if the second write is small and done before the ACK from the other end nagle holds on to the data briefly in the hope you are going to write more. Thanks to the nice people on the libcurl mailing list the problem was identified as formadd does a multipart form and internally in libcurl this results in 2 writes to the socket. It appears there is a way to do a simple POST like this:

and this results in a single write to the socket and hence no nagle \o/

Benchmarking again I now get:

Benchmark: timing 1000 iterations of curl, lwp...
curl: 17 wallclock secs ( 0.36 usr + 0.08 sys = 0.44 CPU) @ 2272.73/s (n=1000)
lwp: 22 wallclock secs ( 4.14 usr + 0.71 sys = 4.85 CPU) @ 206.19/s (n=1000)

Finally there was the issue of cookies. In LWP using and storing cookies to a file is simple:

As I'm using the same LWP::UserAgent for all GET and POST requests and they are to the same domain I can issue a POST to login and follow it with a GET (which needs the cookie) and all is well, LWP just does the right thing. However, with Curl I'm using a separate WWW::Curl::Easy object for GETing and POSTing because I never found a way of resetting the object method so the cookie obtained in the POST is not available in the following GETs. Fortunately, Curl has a way of sharing cookies so I ended up with something like the following in my own wrapper:

and a XXX::Protocol::Curl's as something like:

So in addition to getting a speedup from using libcurl my application code hardly changes since I return an HTTP::Response for both LWP and Curl.

Now when I profile the code using Curl the HTTP parts barely register and my application is loads faster.

Update I saw in the Perl weekly at September 19, 2011 Gabor Szabo said "The code is more complex though and I am not sure if being faster on the CPU really matters. After all these both solve network intensive tasks. Waiting for the HTTP request still takes a lot more time than the CPU usage". I would have liked to have answered that but there does not seem to be a way to comment on the Perl weekly. The first thing is I don't know of any computers which only run one process so the less CPU time your process uses the more is available to the other processes. The second thing is that WWW::Curl::Easy is faster in real time than LWP; if you don't think 2000 GETs in 3s instead of 9s is an improvement you need then you don't need to bother using Curl, I do want that improvement.

Update2 When I ran this code with our actuall application the POSTs were still taking longer than LWP even though my test code is the other way around. At the moment I'm still setting CURLOPT_FORBID_REUSE.