How to set TCP keep Alive in java from httpclient

My java application which resides in AWS private subnet connects to an http server via AWS Nat gateway. I am calling a POST request via httpclient to the HTTP server. That request will take more than 10 minutes to complete. I have configured a socket time out and connection timeout of 1 hour as this this a background task . But the intermediate AWS NAT gateway will send back a RST packet after 300 secs [5 mins] and cause the connection to get resetted , there is no way i can increase the NAT gateway timeout. So i need to handle the problem from my application side.

My strategy is to use a tcp keep alive time which will send a packet say every 240 secs to keep the connection active. I have configured this
as below

CloseableHttpClient httpClient = HttpClients.createDefault()

HttpParams params = httpClient.getParams();

HttpConnectionParams.setConnectionTimeout(params, 3600000); //connection Timeout

HttpConnectionParams.setSoTimeout(params, 3600000); // Socket Time out

HttpConnectionParams.setSoKeepalive(params, true); //Enable Socket level keep alive time

and then call the post request via execute method

HttpPost post = new HttpPost("http://url");

HttpResponse response = httpClient.execute(post);

Since i am using a linux system i have configured the server with following Sysctl values

sysctl -w net.ipv4.tcp_keepalive_time=240 

sysctl -w net.ipv4.tcp_keepalive_intvl=240

sysctl -w net.ipv4.tcp_keepalive_probes=10

But while executing the program the keep alive is not enabled and connection fails as previous.

I have checked this with netstat -o option and as shown below keep alive is off

tcp        0      0 192.168.1.141:43770     public_ip:80          ESTABLISHED 18134/java           off (0.00/0/0)

Is there any way i can set TCP keep alive from java code using httpclient . Also i can see HttpConnectionParams are deprecated. But I couldnt find any new class which can set keep alive

Thanks

edited Nov 19 '18 at 14:43

asked Nov 19 '18 at 13:00

Syam S

137

Yes i have double checked that . The timeout of the http server is also 1 hr . it is the NAT gateway is causing the issue . I have tried the same post call via curl and with a timeout less than 300 secs the request succeeds . if i give a time out greater than 300 it fails. Also when we are not using the NAT gateway then also request succeeds.
– Syam S
Nov 19 '18 at 13:08

add a comment |

My strategy is to use a tcp keep alive time which will send a packet say every 240 secs to keep the connection active. I have configured this
as below

CloseableHttpClient httpClient = HttpClients.createDefault()

HttpParams params = httpClient.getParams();

HttpConnectionParams.setConnectionTimeout(params, 3600000); //connection Timeout

HttpConnectionParams.setSoTimeout(params, 3600000); // Socket Time out

HttpConnectionParams.setSoKeepalive(params, true); //Enable Socket level keep alive time

and then call the post request via execute method

HttpPost post = new HttpPost("http://url");

HttpResponse response = httpClient.execute(post);

Since i am using a linux system i have configured the server with following Sysctl values

sysctl -w net.ipv4.tcp_keepalive_time=240 

sysctl -w net.ipv4.tcp_keepalive_intvl=240

sysctl -w net.ipv4.tcp_keepalive_probes=10

But while executing the program the keep alive is not enabled and connection fails as previous.

I have checked this with netstat -o option and as shown below keep alive is off

tcp        0      0 192.168.1.141:43770     public_ip:80          ESTABLISHED 18134/java           off (0.00/0/0)

Is there any way i can set TCP keep alive from java code using httpclient . Also i can see HttpConnectionParams are deprecated. But I couldnt find any new class which can set keep alive

Thanks

edited Nov 19 '18 at 14:43

asked Nov 19 '18 at 13:00

Syam S

137

Yes i have double checked that . The timeout of the http server is also 1 hr . it is the NAT gateway is causing the issue . I have tried the same post call via curl and with a timeout less than 300 secs the request succeeds . if i give a time out greater than 300 it fails. Also when we are not using the NAT gateway then also request succeeds.
– Syam S
Nov 19 '18 at 13:08

add a comment |

My strategy is to use a tcp keep alive time which will send a packet say every 240 secs to keep the connection active. I have configured this
as below

CloseableHttpClient httpClient = HttpClients.createDefault()

HttpParams params = httpClient.getParams();

HttpConnectionParams.setConnectionTimeout(params, 3600000); //connection Timeout

HttpConnectionParams.setSoTimeout(params, 3600000); // Socket Time out

HttpConnectionParams.setSoKeepalive(params, true); //Enable Socket level keep alive time

and then call the post request via execute method

HttpPost post = new HttpPost("http://url");

HttpResponse response = httpClient.execute(post);

Since i am using a linux system i have configured the server with following Sysctl values

sysctl -w net.ipv4.tcp_keepalive_time=240 

sysctl -w net.ipv4.tcp_keepalive_intvl=240

sysctl -w net.ipv4.tcp_keepalive_probes=10

But while executing the program the keep alive is not enabled and connection fails as previous.

I have checked this with netstat -o option and as shown below keep alive is off

tcp        0      0 192.168.1.141:43770     public_ip:80          ESTABLISHED 18134/java           off (0.00/0/0)

Is there any way i can set TCP keep alive from java code using httpclient . Also i can see HttpConnectionParams are deprecated. But I couldnt find any new class which can set keep alive

Thanks

edited Nov 19 '18 at 14:43

asked Nov 19 '18 at 13:00

Syam S

137

My strategy is to use a tcp keep alive time which will send a packet say every 240 secs to keep the connection active. I have configured this
as below

CloseableHttpClient httpClient = HttpClients.createDefault()

HttpParams params = httpClient.getParams();

HttpConnectionParams.setConnectionTimeout(params, 3600000); //connection Timeout

HttpConnectionParams.setSoTimeout(params, 3600000); // Socket Time out

HttpConnectionParams.setSoKeepalive(params, true); //Enable Socket level keep alive time

and then call the post request via execute method

HttpPost post = new HttpPost("http://url");

HttpResponse response = httpClient.execute(post);

Since i am using a linux system i have configured the server with following Sysctl values

sysctl -w net.ipv4.tcp_keepalive_time=240 

sysctl -w net.ipv4.tcp_keepalive_intvl=240

sysctl -w net.ipv4.tcp_keepalive_probes=10

But while executing the program the keep alive is not enabled and connection fails as previous.

I have checked this with netstat -o option and as shown below keep alive is off

tcp        0      0 192.168.1.141:43770     public_ip:80          ESTABLISHED 18134/java           off (0.00/0/0)

Is there any way i can set TCP keep alive from java code using httpclient . Also i can see HttpConnectionParams are deprecated. But I couldnt find any new class which can set keep alive

Thanks

java apache-httpclient-4.x tcp-keepalive

edited Nov 19 '18 at 14:43

asked Nov 19 '18 at 13:00

Syam S

137

edited Nov 19 '18 at 14:43

asked Nov 19 '18 at 13:00

Syam S

137

edited Nov 19 '18 at 14:43

asked Nov 19 '18 at 13:00

Syam S

137

asked Nov 19 '18 at 13:00

Syam S

137

asked Nov 19 '18 at 13:00

Syam S

137

Yes i have double checked that . The timeout of the http server is also 1 hr . it is the NAT gateway is causing the issue . I have tried the same post call via curl and with a timeout less than 300 secs the request succeeds . if i give a time out greater than 300 it fails. Also when we are not using the NAT gateway then also request succeeds.
– Syam S
Nov 19 '18 at 13:08

add a comment |

Yes i have double checked that . The timeout of the http server is also 1 hr . it is the NAT gateway is causing the issue . I have tried the same post call via curl and with a timeout less than 300 secs the request succeeds . if i give a time out greater than 300 it fails. Also when we are not using the NAT gateway then also request succeeds.
– Syam S
Nov 19 '18 at 13:08

Yes i have double checked that . The timeout of the http server is also 1 hr . it is the NAT gateway is causing the issue . I have tried the same post call via curl and with a timeout less than 300 secs the request succeeds . if i give a time out greater than 300 it fails. Also when we are not using the NAT gateway then also request succeeds.
– Syam S
Nov 19 '18 at 13:08

add a comment |

2 Answers
2

active

oldest

votes

I have found a solution to the problem . Curious case is there is no way i can use some builder class in httpclient to pass socket keep alive . One method as i specified in the question is using HttpConnectionParams as below but this is not working and this class is now deprecated.

HttpParams params = httpClient.getParams();

HttpConnectionParams.setSoKeepalive(params, true);

So while checking apache http docs I can see that now connection parameters are passed to httpclient via RequestConfig class . Builders of this class provide solution to set connection_time_out and socket_time_out. But checking the socurce code of this I couldnt see an option to enable SocketKeepAlive which is what we want. So the only solution is directly creating a Socket using SocketBuilder class and pass that to the HttpClientBuilder.

Following is the working code

SocketConfig socketConfig = SocketConfig.custom().setSoKeepAlive(true).setSoTimeout(3600000).build(); //We need to set socket keep alive

        RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(3600000).build();

        CloseableHttpClient httpClient = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).

                                           setDefaultSocketConfig(socketConfig).build();

HttpPost post = new HttpPost(url.toString());

HttpResponse response = httpClient.execute(post);

While executing above i can see that keep alive is properly set in the socket based on the sysctl values i set in linux kernel

tcp        0      0 localip:48314     public_ip:443     ESTABLISHED 14863/java          keepalive (234.11/0/0)

If some one has a better solution to enable Socket Keep alive from Requestconfig class or any other high level builder class i am open to suggestions.

answered Nov 20 '18 at 7:44

Syam S

137

add a comment |

Keeping an HTTP connection open but inactive for a long period is a bad design choice. HTTP is a request-response protocol, with the implication that requests and responses are quick.

Holding a connection open holds resources. From the perspective of the server (and network firewalls and routers) a client that opens a connection and begins a request (A POST in your case) but does not send any bytes for a long period is indistinguishable from a client that will never send any more data, because it is faulty or malicious (conducting a DOS attack). The server (and network hardware) is right to conclude that the right thing to do is to shutdown the connection and reclaim the resources used for it. You are trying to fight against correct behaviour that occurs for good reasons. Even if you manage to workaround the TCP shutdown you will find other problems, such as HTTP server timeouts and database timeouts.

You should instead be reconsidered the design of communication between the two components. That is, this looks like an XY Problem. You might consider

Having the client wait until it has a complete upload to perform before starting the POST.

Splitting the uploads into smaller, more frequent uploads.

Use a protocol other than HTTP.

answered Nov 20 '18 at 9:43

Raedwald

25.8k2295156

Yes i agree with you in the sense that HTTP requests and responses must be completed fast . But in our case there are some tasks that will take time to complete. We are moving these tasks to a listener based model via rabbitmq . But for now i need to stick with this method.
– Syam S
Nov 20 '18 at 10:02

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53375222%2fhow-to-set-tcp-keep-alive-in-java-from-httpclient%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

2 Answers
2

active

oldest

votes

2 Answers
2

active

oldest

votes

HttpParams params = httpClient.getParams();

HttpConnectionParams.setSoKeepalive(params, true);

Following is the working code

SocketConfig socketConfig = SocketConfig.custom().setSoKeepAlive(true).setSoTimeout(3600000).build(); //We need to set socket keep alive

        RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(3600000).build();

        CloseableHttpClient httpClient = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).

                                           setDefaultSocketConfig(socketConfig).build();

HttpPost post = new HttpPost(url.toString());

HttpResponse response = httpClient.execute(post);

While executing above i can see that keep alive is properly set in the socket based on the sysctl values i set in linux kernel

tcp        0      0 localip:48314     public_ip:443     ESTABLISHED 14863/java          keepalive (234.11/0/0)

If some one has a better solution to enable Socket Keep alive from Requestconfig class or any other high level builder class i am open to suggestions.

answered Nov 20 '18 at 7:44

Syam S

137

add a comment |

HttpParams params = httpClient.getParams();

HttpConnectionParams.setSoKeepalive(params, true);

Following is the working code

SocketConfig socketConfig = SocketConfig.custom().setSoKeepAlive(true).setSoTimeout(3600000).build(); //We need to set socket keep alive

        RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(3600000).build();

        CloseableHttpClient httpClient = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).

                                           setDefaultSocketConfig(socketConfig).build();

HttpPost post = new HttpPost(url.toString());

HttpResponse response = httpClient.execute(post);

While executing above i can see that keep alive is properly set in the socket based on the sysctl values i set in linux kernel

tcp        0      0 localip:48314     public_ip:443     ESTABLISHED 14863/java          keepalive (234.11/0/0)

If some one has a better solution to enable Socket Keep alive from Requestconfig class or any other high level builder class i am open to suggestions.

answered Nov 20 '18 at 7:44

Syam S

137

add a comment |

HttpParams params = httpClient.getParams();

HttpConnectionParams.setSoKeepalive(params, true);

Following is the working code

SocketConfig socketConfig = SocketConfig.custom().setSoKeepAlive(true).setSoTimeout(3600000).build(); //We need to set socket keep alive

        RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(3600000).build();

        CloseableHttpClient httpClient = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).

                                           setDefaultSocketConfig(socketConfig).build();

HttpPost post = new HttpPost(url.toString());

HttpResponse response = httpClient.execute(post);

While executing above i can see that keep alive is properly set in the socket based on the sysctl values i set in linux kernel

tcp        0      0 localip:48314     public_ip:443     ESTABLISHED 14863/java          keepalive (234.11/0/0)

If some one has a better solution to enable Socket Keep alive from Requestconfig class or any other high level builder class i am open to suggestions.

answered Nov 20 '18 at 7:44

Syam S

137

HttpParams params = httpClient.getParams();

HttpConnectionParams.setSoKeepalive(params, true);

Following is the working code

SocketConfig socketConfig = SocketConfig.custom().setSoKeepAlive(true).setSoTimeout(3600000).build(); //We need to set socket keep alive

        RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(3600000).build();

        CloseableHttpClient httpClient = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).

                                           setDefaultSocketConfig(socketConfig).build();

HttpPost post = new HttpPost(url.toString());

HttpResponse response = httpClient.execute(post);

While executing above i can see that keep alive is properly set in the socket based on the sysctl values i set in linux kernel

tcp        0      0 localip:48314     public_ip:443     ESTABLISHED 14863/java          keepalive (234.11/0/0)

If some one has a better solution to enable Socket Keep alive from Requestconfig class or any other high level builder class i am open to suggestions.

answered Nov 20 '18 at 7:44

Syam S

137

answered Nov 20 '18 at 7:44

Syam S

137

answered Nov 20 '18 at 7:44

Syam S

137

answered Nov 20 '18 at 7:44

Syam S

137

add a comment |

Keeping an HTTP connection open but inactive for a long period is a bad design choice. HTTP is a request-response protocol, with the implication that requests and responses are quick.

You should instead be reconsidered the design of communication between the two components. That is, this looks like an XY Problem. You might consider

Having the client wait until it has a complete upload to perform before starting the POST.

Splitting the uploads into smaller, more frequent uploads.

Use a protocol other than HTTP.

answered Nov 20 '18 at 9:43

Raedwald

25.8k2295156

Yes i agree with you in the sense that HTTP requests and responses must be completed fast . But in our case there are some tasks that will take time to complete. We are moving these tasks to a listener based model via rabbitmq . But for now i need to stick with this method.
– Syam S
Nov 20 '18 at 10:02

add a comment |

Keeping an HTTP connection open but inactive for a long period is a bad design choice. HTTP is a request-response protocol, with the implication that requests and responses are quick.

You should instead be reconsidered the design of communication between the two components. That is, this looks like an XY Problem. You might consider

Having the client wait until it has a complete upload to perform before starting the POST.

Splitting the uploads into smaller, more frequent uploads.

Use a protocol other than HTTP.

answered Nov 20 '18 at 9:43

Raedwald

25.8k2295156

Yes i agree with you in the sense that HTTP requests and responses must be completed fast . But in our case there are some tasks that will take time to complete. We are moving these tasks to a listener based model via rabbitmq . But for now i need to stick with this method.
– Syam S
Nov 20 '18 at 10:02

add a comment |

Keeping an HTTP connection open but inactive for a long period is a bad design choice. HTTP is a request-response protocol, with the implication that requests and responses are quick.

You should instead be reconsidered the design of communication between the two components. That is, this looks like an XY Problem. You might consider

Having the client wait until it has a complete upload to perform before starting the POST.

Splitting the uploads into smaller, more frequent uploads.

Use a protocol other than HTTP.

answered Nov 20 '18 at 9:43

Raedwald

25.8k2295156

Keeping an HTTP connection open but inactive for a long period is a bad design choice. HTTP is a request-response protocol, with the implication that requests and responses are quick.

You should instead be reconsidered the design of communication between the two components. That is, this looks like an XY Problem. You might consider

Having the client wait until it has a complete upload to perform before starting the POST.

Splitting the uploads into smaller, more frequent uploads.

Use a protocol other than HTTP.

answered Nov 20 '18 at 9:43

Raedwald

25.8k2295156

answered Nov 20 '18 at 9:43

Raedwald

25.8k2295156

answered Nov 20 '18 at 9:43

Raedwald

25.8k2295156

answered Nov 20 '18 at 9:43

Raedwald

25.8k2295156

Yes i agree with you in the sense that HTTP requests and responses must be completed fast . But in our case there are some tasks that will take time to complete. We are moving these tasks to a listener based model via rabbitmq . But for now i need to stick with this method.
– Syam S
Nov 20 '18 at 10:02

add a comment |

Yes i agree with you in the sense that HTTP requests and responses must be completed fast . But in our case there are some tasks that will take time to complete. We are moving these tasks to a listener based model via rabbitmq . But for now i need to stick with this method.
– Syam S
Nov 20 '18 at 10:02

Yes i agree with you in the sense that HTTP requests and responses must be completed fast . But in our case there are some tasks that will take time to complete. We are moving these tasks to a listener based model via rabbitmq . But for now i need to stick with this method.
– Syam S
Nov 20 '18 at 10:02

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

Some of your past answers have not been well-received, and you're in danger of being blocked from answering.

Please pay close attention to the following guidance:

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu