HttpClient and CLOSE_WAIT

Today, I was running some load scripts on code that executed an HTTP POST request to a REST service. The load test essentially ran the HTTP call over and over. When the client was done running, I ended up with a bunch of TCP connections in CLOSE_WAIT status. To see the TCP connection states, run the netstat command on Unix (I’m guessing Windows has something similar). While the system appears to be running fine, CLOSE_WAIT means that the socket is inactive but still allocated. As the system continues to run and sockets in this state accumulate, the OS will eventually run out of sockets and everything will slow down to a crawl until it ultimately implodes and everything collapses.

I’ve seen this problem before so it didn’t take long to figure out, but I figured I’d post this to help those who haven’t seen it.

First, a quick TCP refresher… When a TCP connection is about to close, its finalization is negotiated by both parties. Think of it as breaking a contract in a civilized manner. Both parties sign the paper and it’s all good. In geek talk, this is done via the FIN/ACK messages. Party A sends a FIN message to indicate it wants to close the socket. Party B sends an ACK saying it received the message and is considering the demand. Party B then cleans up and sends a FIN to Party A. Party A responds with the ACK and everyone walks away:

A --FIN-> B
A <-ACK-- B
A <-FIN-- B
A --ACK-> B

The problem comes in when B doesn’t send its FIN. A is kinda stuck waiting for it. It has initiated its finalization sequence and is waiting for the other party to do the same.

In the case of an HTTP exchange, the same thing can happen if not managed correctly. When using Apache HttpClient, a connection isn’t closed unless the code explicitly closes it. Unfortunately it’s not completely obvious how to close the connection from the client’s perspective:

HttpResponse response = httpClient.execute(postMethod);

It is even more problematic if you are using a connection manager which will create a connection pool. Also note that it’s not enough to call releaseConnection() to close the connection. This is the most common mistake.

What works best in my opinion is to reverse the sequence and let the server initiate the finalization. This means that we want to let the server send the initial FIN. This way the client code doesn’t need to do anything specific to manage the connection. We just let the TCP implementation take care of responding. Okay, so how do we do that? Simple… The HTTP protocol defines a header parameter called “Connection” which supports the “close” option.

RFC 2616, 14.10

In your code, before calling the execute method, add:
postMethod.addHeader("Connection", "close");

This header parameter will tell the server that you’ll want it to close the connection when it has sent the response. Your client socket will receive a FIN from the server, and the OS’ TCP implementation will do its own cleanup.

CLOSE_WAIT gone… Moving on…