Apache HttpClient socket超时配置和原理

背景

最近因为工作原理，需要研究一下各种 client 的 read timeout 的配置和原理
包括 Apache HttpClient 和 jedis
简单分析了一下，把结论分享在这里

（注意，只是 read timeout, 还有很多 timeout, 这里不展开研究）

Apache HttpClient timeout 配置

这里贴一个常用的工具类

import org.apache.http.client.HttpClient;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.impl.client.HttpClientBuilder;

public class HttpClientUtils {
        
    public static final String DEFAULT_CHARSET = "UTF-8";
    
    //默认值参数在 org.apache.http.impl.conn.PoolingHttpClientConnectionManager.PoolingHttpClientConnectionManager
    //设置最大连接数,默认值是 20
    private static final int MAX_CONN_TOTAL = 800;   
    
    //每个路由最大连接数(一个 host:port就是一个连接)，默认 2
    private static final int MAX_CONN_PER_ROUTE = 400;   
    
    private static final int CONNECTION_POOL_TIMEOUT = 1000;   //从连接池中获取连接的超时时间, 1000毫秒
    
    private static final int CONNECTION_TIMEOUT = 1000;       //与服务器连接超时, 1 秒
    private static final int SOCKET_TIMEOUT = 10 * 1000;           //读取服务器返回的数据 超时, 10 秒
    
    /**
     * <br>默认提供一个 client
     */
    public static final HttpClient DEFAULT_CLIENT;
    
    static {
        RequestConfig clusterConfig = RequestConfig.custom()
                .setConnectionRequestTimeout(CONNECTION_POOL_TIMEOUT)
                .setConnectTimeout(CONNECTION_TIMEOUT)               // 与服务器连接超时
                .setSocketTimeout(SOCKET_TIMEOUT)                // 读取服务器返回的数据 超时
                .build();
        
        //参数含义参考 下面两个博客
        //https://blog.csdn.net/shootyou/article/details/6615051
        //https://blog.csdn.net/shootyou/article/details/6415248
        
        DEFAULT_CLIENT = HttpClientBuilder.create()
            .setDefaultRequestConfig(clusterConfig)
            .setMaxConnTotal(MAX_CONN_TOTAL)
            .setMaxConnPerRoute(MAX_CONN_PER_ROUTE)
            .build();
    }

}

可以看到 read timeout 其实就是 socket timeout， 也就是这行代码

.setSocketTimeout(SOCKET_TIMEOUT)                // 读取服务器返回的数据 超时

原理

那么是怎么实现的呢？
我们写一个接口，阻塞 11秒， 看下堆栈就知道了

java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:84)
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)

Native Method 我们看不到代码，那去看下 java.net.SocketInputStream.socketRead(SocketInputStream.java:116)

// wrap native call to allow instrumentation
/**
 * Reads into an array of bytes at the specified offset using
 * the received socket primitive.
 * @param fd the FileDescriptor
 * @param b the buffer into which the data is read
 * @param off the start offset of the data
 * @param len the maximum number of bytes read
 * @param timeout the read timeout in ms
 * @return the actual number of bytes read, -1 is
 *          returned when the end of the stream is reached.
 * @exception IOException If an I/O error has occurred.
 */
private int socketRead(FileDescriptor fd,
                       byte b[], int off, int len,
                       int timeout)
    throws IOException {
    return socketRead0(fd, b, off, len, timeout);
}

可以看到，是jdk 提供了timeout参数，实现了这个超时功能， HttpClient 只是使用了这个特性

JedisPool

配置

我们看下 redis.clients.jedis.JedisPool 的构造方法源码

下面是一个构造 JedisPool 的样例

private static JedisPool initJedisPool(String url) {
        JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
        jedisPoolConfig.setMaxTotal(200);
        jedisPoolConfig.setMaxIdle(50);
        jedisPoolConfig.setMaxWaitMillis(10000L);
        jedisPoolConfig.setTestOnBorrow(true);

        return new JedisPool(jedisPoolConfig, URI.create(url));
    }

看下这个两个参数的构造方法

public JedisPool(final GenericObjectPoolConfig poolConfig, final URI uri) {
    this(poolConfig, uri, Protocol.DEFAULT_TIMEOUT);
}

这个常量是下面这样的，也就是2秒

public static final int DEFAULT_TIMEOUT = 2000;

再继续看下三个参数的构造方法

public JedisPool(final GenericObjectPoolConfig poolConfig, final URI uri, final int timeout) {
    this(poolConfig, uri, timeout, timeout);
}

public JedisPool(final GenericObjectPoolConfig poolConfig, final URI uri,
  final int connectionTimeout, final int soTimeout) {
    super(poolConfig, new JedisFactory(uri, connectionTimeout, soTimeout, null, false,
    null, null, null));
}

可以看到，Protocol.DEFAULT_TIMEOUT 被设置到两个值里面去了
分别是 connectionTimeout 和 soTimeout

soTimeout 应该全称是 socket timeout

原理

我们也试着让 redis 慢点返回，看看异常堆栈

redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
    at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:202)
    at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40)
    at redis.clients.jedis.Protocol.process(Protocol.java:153)
    at redis.clients.jedis.Protocol.read(Protocol.java:218)
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:341)
    at redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:260)
    at redis.clients.jedis.Connection.getBulkReply(Connection.java:249)
    at redis.clients.jedis.Jedis.get(Jedis.java:156)
    
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.net.SocketInputStream.read(SocketInputStream.java:127)
    at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:196)
    ... 14 common frames omitted

可以看到超时的原理差不多，都是jdk提供的特性