We Keep Coding

HTTPClient hung or slow in multi-threaded environment

I am using following version of httpclient api to call the rest webservice.
httpcore4.4.9
httpclient 4.5.9
I am using the ExecutorService to execute the task where it is calling my following methods to get the data from rest API. ExecutorService execute about 30+ rest call in one transaction and i observed that, httpclient HUNG or SLOW for few of the rest call and it hit the performance issue. Can you please check if below i am using httpclient correct way ?
String output = null;
CloseableHttpClient httpClient = getHttpClients();
try (CloseableHttpResponse response = httpClient.execute(httpGet)) {
HttpEntity entity = response.getEntity();
output = EntityUtils.toString(entity);
EntityUtils.consume(response.getEntity());
} catch (Exception e) {
log.error("error during call to REST API " + httpGet.getURI() + " ", e);
} finally {
httpClient.close();
httpGet.releaseConnection();
httpGet.abort();
}
return output;
}
private CloseableHttpClient getHttpClients() {
long startTime = System.currentTimeMillis();
long endTime = 0;
int timeout = 12000;
RequestConfig.Builder requestBuilder = RequestConfig.custom();
requestBuilder.setConnectTimeout(timeout);
requestBuilder.setConnectionRequestTimeout(timeout);
SocketConfig socketConfig = SocketConfig.custom().setSoTimeout(timeout).build();
HttpClientBuilder builder = HttpClientBuilder.create();
builder.setDefaultRequestConfig(requestBuilder.build());
builder.disableContentCompression();
builder.setDefaultSocketConfig(socketConfig);
CloseableHttpClient httpClient = builder
.build();
endTime = System.currentTimeMillis();
log.debug("Total time took to build client = " + (endTime - startTime));
return httpClient;
} ```

As per my experience, this problem is occurring because of EntityUtils.consume(response.getEntity()) .
You could find what it actually does here.
Try removing and running your code to match the speed and decide if you want it or not.
Below is the code used by me:
CloseableHttpClient httpClient = HttpClients.createDefault();
try {
HttpGet request = new HttpGet(url);
CloseableHttpResponse response = httpClient.execute(request);
String responseString = EntityUtils.toString(response.getEntity());
} catch (Exception ex) {
log.error(ex.toString());
}

Try to adjust MaxTotal and MaxPerRoute variable in PoolingHttpClientConnectionManager class.
connectionManager.setMaxTotal(maxTotalConnections);
connectionManager.setDefaultMaxPerRoute(maxConnectionsPerRoute);

Java socket is closed - When posting SOAP request

first time posting a question on this site. I've been working on a project where I have to generate a signed(Header x509) SOAP request and submit it to an https URL where the webservice resides. Thus far I've been able to generate the Signed SOAP request, but I'm getting a socket is closed exception when I post the request. Can anyone help me figure out what is my code missing that could be causing this error. Thank you in advance!
Below is my code for how I'm posting the request, and the error:
SSLContext sslcontext = SSLContexts.custom()
.loadTrustMaterial(new File("my keystore"), "mypassword".toCharArray(),
new TrustSelfSignedStrategy())
.build();
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(
sslcontext,
new String[] { "TLSv1" },
null,
SSLConnectionSocketFactory.getDefaultHostnameVerifier());
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(300 * 1000)
.setConnectTimeout(300 * 1000)
.setConnectionRequestTimeout(300 * 1000)
.build();
HttpClient httpclient = HttpClients.custom() //CloseableHttpClient
.setDefaultRequestConfig(requestConfig)
.setSSLSocketFactory(sslsf)
.build();
HttpPost httpPost;
httpPost = new HttpPost("https://my webservice");
httpPost.setHeader("Content-type", "test/xml");
File input = new File(esbXmlFile);
InputStream inputStream = new FileInputStream(input);
InputStreamEntity inputStreamEntity = new InputStreamEntity(inputStream);
httpPost.setEntity(inputStreamEntity);
String content = null;
int statusCode = 1; //statusCode is response code form WS.
try
{
HttpResponse response = httpclient.execute(httpPost);
statusCode = response.getStatusLine().getStatusCode();
System.out.println("statusCode : " + statusCode);
HttpEntity entity = response.getEntity();
content = EntityUtils.toString(entity);
}
catch (SocketException e) {
e.getStackTrace();
}
catch (IOException e) {
e.getStackTrace();
}
I get this error:
DEBUG (main) [org.apache.http.conn.ssl.SSLConnectionSocketFactory] - Starting handshake
2016-11-28 16:24:04,019 DEBUG (main) [org.apache.http.impl.conn.DefaultManagedHttpClientConnection] - http-outgoing-1: Shutdown connection
2016-11-28 16:24:04,019 DEBUG (main) [org.apache.http.impl.execchain.MainClientExec] - Socket is closed
java.net.SocketException: Socket is closed
at java.net.Socket.setSoLinger(Socket.java:986)
at org.apache.http.impl.BHttpConnectionBase.shutdown(BHttpConnectionBase.java:305)
at org.apache.http.impl.conn.DefaultManagedHttpClientConnection.shutdown(DefaultManagedHttpClientConnection.java:97)
at org.apache.http.impl.conn.LoggingManagedHttpClientConnection.shutdown(LoggingManagedHttpClientConnection.java:89)
at org.apache.http.impl.conn.CPoolEntry.shutdownConnection(CPoolEntry.java:74)
at org.apache.http.impl.conn.CPoolProxy.shutdown(CPoolProxy.java:96)
at org.apache.http.impl.execchain.ConnectionHolder.abortConnection(ConnectionHolder.java:127)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:352)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)

Maybe you have to set:
httpPost.setHeader("Content-type", "test/xml");
into
httpPost.setHeader("Content-type", "text/xml");
difference is test/xml into text/xml

Httpcomponents httpcore and httpclient HTTP_status 400

I am trying to do:
Send request from HttpClient (based on HttpComponents HttpClient 4.5).
Receive that request in HttpServer (based on HttpComponents HttpCore 4.4.1).
HttpServer must answer to HttpClient with different HttpStatus codes and string entities as body.
Problem: If HttpServer make answer with status code 200 (or any others, not checked) then it is worked fine and no exceptions on server side. But if server set answer status code 400, then there is IOException on HttpServer has been occured. Description on russian is "Удаленный хост принудительно разорвал существующее подключение", on english i think it is "Client closed connection". Simple one: on status 200 there is no problem, on 400 it is exception occured on server.
Exception string:
java.io.IOException: Удаленный хост принудительно разорвал существующее подключение
at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[na:1.7.0_51]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) ~[na:1.7.0_51]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.7.0_51]
at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.7.0_51]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[na:1.7.0_51]
at org.apache.http.nio.reactor.ssl.SSLIOSession.receiveEncryptedData(SSLIOSession.java:449) ~[httpcore-nio-4.4.1.jar:4.4.1]
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:503) ~[httpcore-nio-4.4.1.jar:4.4.1]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:122) ~[httpcore-nio-4.4.1.jar:4.4.1]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164) [httpcore-nio-4.4.1.jar:4.4.1]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339) [httpcore-nio-4.4.1.jar:4.4.1]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317) [httpcore-nio-4.4.1.jar:4.4.1]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278) [httpcore-nio-4.4.1.jar:4.4.1]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) [httpcore-nio-4.4.1.jar:4.4.1]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590) [httpcore-nio-4.4.1.jar:4.4.1]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
HttpServer code:
HttpProcessor httpproc = HttpProcessorBuilder.create()
.add(new ResponseDate())
.add(new ResponseServer("HTTP/1.1 WTX Server"))
.add(new ResponseContent())
.add(new ResponseConnControl()).build();
UriHttpAsyncRequestHandlerMapper reqistry = new UriHttpAsyncRequestHandlerMapper();
reqistry.register("*", new HttpServerURLHandler());
HttpAsyncService protocolHandler = new HttpServerConnectionsHandler(httpproc, reqistry);
try {
String keyStoreFile = Config.getString("HTTPServer.keyStoreFile");
String keyStoreFilePassword = Config.getString("HTTPServer.keyStoreFilePassword");
FileInputStream fin = new FileInputStream(keyStoreFile);
KeyStore keystore = KeyStore.getInstance("jks");
keystore.load(fin, keyStoreFilePassword.toCharArray());
KeyManagerFactory kmfactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
kmfactory.init(keystore, keyStoreFilePassword.toCharArray());
KeyManager[] keymanagers = kmfactory.getKeyManagers();
SSLContext sslcontext = SSLContext.getInstance("TLS");
sslcontext.init(keymanagers, null, null);
NHttpConnectionFactory<DefaultNHttpServerConnection> connFactory = new SSLNHttpServerConnectionFactory(sslcontext, null, ConnectionConfig.DEFAULT);
IOEventDispatch ioEventDispatch = new DefaultHttpServerIODispatch(protocolHandler, connFactory);
IOReactorConfig config = IOReactorConfig.custom()
//.setIoThreadCount(10)
//.setSoTimeout(5000)
//.setConnectTimeout(4000)
//.setSoKeepAlive(true)
//.setSoReuseAddress(true)
//.setRcvBufSize(65535)
//.setTcpNoDelay(true)
.build();
ListeningIOReactor ioReactor = new DefaultListeningIOReactor(config);
ioReactor.listen(new InetSocketAddress(socketAddr, socketPort));
ioReactor.execute(ioEventDispatch);
} catch (Exception e) {
MDC.put(ApplicationInit.LOGGERVAR, ApplicationInit.LOGGERCTX.HTTPSERVER.toString());
logger.error("Error while creating HTTP Server instance.", e);
}
URL Hander code:
public class HttpServerURLHandler implements HttpAsyncRequestHandler<HttpRequest> {
public static final Logger logger = LoggerFactory.getLogger(HttpServerURLHandler.class);
private BasicHttpResponse httpResponse = null;
public HttpServerURLHandler() {
super();
}
public HttpAsyncRequestConsumer<HttpRequest> processRequest(final HttpRequest request, final HttpContext context) {
return new BasicAsyncRequestConsumer();
}
public void handle(final HttpRequest httpRequest, final HttpAsyncExchange httpExchange, final HttpContext httpContext) throws HttpException, IOException {
String string1 = "ABCDEFGHIJKLMNOPQRSTUVWXYZ++++++++++++++++++++++++++++++++++++++++++++++++abcdefghijklmnopqrstuvwxyz";
string1 += "ABCDEFGHIJKLMNOPQRSTUVWXYZ++++++++++++++++++++++++++++++++++++++++++++++++abcdefghijklmnopqrstuvwxyz";
int httpCode = 400;
String httpCodeString = EnglishReasonPhraseCatalog.INSTANCE.getReason(httpCode, Locale.ENGLISH);
BasicHttpResponse httpResponse = new BasicHttpResponse(HttpVersion.HTTP_1_1, httpCode, httpCodeString);
NStringEntity answerEntity = new NStringEntity(stringXML, Consts.UTF_8);
httpResponse.setEntity(answerEntity);
httpExchange.submitResponse(new BasicAsyncResponseProducer(httpResponse));
}
}
Client code:
RequestConfig config = RequestConfig.custom()
.setConnectTimeout(20000)
.setConnectionRequestTimeout(20000)
.setSocketTimeout(20000)
.build();
SSLContext sslContext = null;
try {
TrustStrategy trustStrategy = new TrustStrategy() {
public boolean isTrusted(X509Certificate[] arg0, String arg1) {
return true;
}
};
sslContext = new SSLContextBuilder().loadTrustMaterial(null, trustStrategy).build();
} catch (Exception e) {
MDC.put(ApplicationInit.LOGGERVAR, ApplicationInit.LOGGERCTX.HTTPCLIENT.toString());
logger.error("Error while creating SSL context for making HTTP request", e);
}
CloseableHttpClient client = HttpClientBuilder.create()
.setDefaultRequestConfig(config)
.setSSLContext(sslContext)
.setSSLHostnameVerifier(new NoopHostnameVerifier())
.build();
String stringURL = "https://serverhost:port/";
try {
HttpPost post = new HttpPost(stringURL);
post.setEntity(httpClientRequest.getEntity());
CloseableHttpResponse httpResponse = client.execute(post);
// Consume entity code
HttpEntity responseEntity = httpResponse.getEntity();
String stringXMLAnswer = EntityUtils.toString(responseEntity);
EntityUtils.consume(responseEntity);
// Some next operations with responseEntity
} catch (Exception e) {
MDC.put(ApplicationInit.LOGGERVAR, ApplicationInit.LOGGERCTX.HTTPCLIENT.toString());
logger.error("Error while make request.", e);
} finally {
try {
// Closing connection
client.close();
} catch (Exception e) {
MDC.put(ApplicationInit.LOGGERVAR, ApplicationInit.LOGGERCTX.HTTPCLIENT.toString());
logger.error("Error while closing connection after making request", e);
}
}

The connection reset is likely to be caused by HTTPCLIENT-1655. Please try the latest 4.5.x snapshot and see if that fixes the problem.

I was find org.apache.http.protocol.ResponseConnControl with code
if (status == HttpStatus.SC_BAD_REQUEST ||
status == HttpStatus.SC_REQUEST_TIMEOUT ||
status == HttpStatus.SC_LENGTH_REQUIRED ||
status == HttpStatus.SC_REQUEST_TOO_LONG ||
status == HttpStatus.SC_REQUEST_URI_TOO_LONG ||
status == HttpStatus.SC_SERVICE_UNAVAILABLE ||
status == HttpStatus.SC_NOT_IMPLEMENTED) {
response.setHeader(HTTP.CONN_DIRECTIVE, HTTP.CONN_CLOSE);
return;
}
My problem reproduced by this status codes.

How to force Apache HttpClient / HttpConnection to absolutely close tcp connection?

I use lot of http connection with Apache HttpClient (org.apache.httpcomponents 4.3.6) to test servers and I cannot force connections to close, even when I schedule HttpConnectionManager to httpClient.getConnectionManager().shutdown(); after 10 seconds in another thread.
httpClient.close() also doesn't help.
Connections can last for minutes or even hours.
I have tried custom SocketConfig and this also not helps:
SocketConfig socketConfig = SocketConfig.custom()
.setSoKeepAlive(false)
.setSoLinger(5)
.setSoReuseAddress(true)
.setSoTimeout(5000)
.setTcpNoDelay(true).build();
The way I am fetching the content:
HttpUriRequest request = new HttpGet(url);
HttpResponse response = httpClient.execute(request);
try (InputStream in = response.getEntity().getContent()) {
String result = IOUtils.toString(in, StandardCharsets.UTF_8);
httpClient.close();
return result;
}
The way I am building HTTP Client:
SocketConfig socketConfig = SocketConfig.custom()
.setSoKeepAlive(false)
.setSoLinger(configuration.getInt("proxy.list.socket.so.linger"))
.setSoReuseAddress(true)
.setSoTimeout(configuration.getInt("proxy.list.socket.so.timeout"))
.setTcpNoDelay(true).build();
HttpClientBuilder builder = HttpClientBuilder.create();
builder.disableAutomaticRetries();
builder.disableContentCompression();
builder.disableCookieManagement();
builder.disableRedirectHandling();
builder.setConnectionReuseStrategy(new NoConnectionReuseStrategy());
builder.setDefaultSocketConfig(socketConfig);
One of my prototypes of doing shutdown:
shutdownExecutor.schedule(() -> {
httpClient.getConnectionManager().closeExpiredConnections();
httpClient.getConnectionManager().closeIdleConnections(1, TimeUnit.SECONDS);
httpClient.getConnectionManager().shutdown();
httpClient.notifyAll();
try {
httpClient.close();
} catch (IOException ex) {
}
}, configuration.getInt("proxy.test.forced.timeout.seconds"), TimeUnit.SECONDS);
String content = HttpContentFetcher.getAndCloseClient(url, httpClient);

RequestConfig has helped. Now it looks like all the connections are discarded in specified limits.
RequestConfig config= RequestConfig.custom()
.setCircularRedirectsAllowed(false)
.setConnectionRequestTimeout(4000)
.setConnectTimeout(4000)
.setMaxRedirects(0)
.setRedirectsEnabled(false)
.setSocketTimeout(4000)
.setStaleConnectionCheckEnabled(true).build();
request.setConfig(config);

Preemptive Basic authentication with Apache HttpClient 4

Is there an easier way to setup the http client for preemptive basic authentication than what described here?
In previous version (3.x) it used to be a simple method call (eg, httpClient.getParams().setAuthenticationPreemptive(true)).
The main thing I want to avoid is adding the BasicHttpContext to each method I execute.

If you are looking to force HttpClient 4 to authenticate with a single request, the following will work:
String username = ...
String password = ...
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(username, password);
HttpRequest request = ...
request.addHeader(new BasicScheme().authenticate(creds, request));

It's difficult to do this without passing a context through every time, but you can probably do it by using a request interceptor. Here is some code that we use (found from their JIRA, iirc):
// Pre-emptive authentication to speed things up
BasicHttpContext localContext = new BasicHttpContext();
BasicScheme basicAuth = new BasicScheme();
localContext.setAttribute("preemptive-auth", basicAuth);
httpClient.addRequestInterceptor(new PreemptiveAuthInterceptor(), 0);
(...)
static class PreemptiveAuthInterceptor implements HttpRequestInterceptor {
public void process(final HttpRequest request, final HttpContext context) throws HttpException, IOException {
AuthState authState = (AuthState) context.getAttribute(ClientContext.TARGET_AUTH_STATE);
// If no auth scheme avaialble yet, try to initialize it
// preemptively
if (authState.getAuthScheme() == null) {
AuthScheme authScheme = (AuthScheme) context.getAttribute("preemptive-auth");
CredentialsProvider credsProvider = (CredentialsProvider) context.getAttribute(ClientContext.CREDS_PROVIDER);
HttpHost targetHost = (HttpHost) context.getAttribute(ExecutionContext.HTTP_TARGET_HOST);
if (authScheme != null) {
Credentials creds = credsProvider.getCredentials(new AuthScope(targetHost.getHostName(), targetHost.getPort()));
if (creds == null) {
throw new HttpException("No credentials for preemptive authentication");
}
authState.setAuthScheme(authScheme);
authState.setCredentials(creds);
}
}
}
}

This is the same solution as Mat's Mannion's, but you don't have to put localContext to each request. It's simpler, but it adds authentication to ALL requests. Useful, if you don't have control over individual requests, as in my case when using Apache Solr, which uses HttpClient internally.
import org.apache.http.HttpException;
import org.apache.http.HttpHost;
import org.apache.http.HttpRequest;
import org.apache.http.HttpRequestInterceptor;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.AuthState;
import org.apache.http.auth.Credentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.impl.auth.BasicScheme;
import org.apache.http.protocol.HttpContext;
import org.apache.http.protocol.HttpCoreContext;
httpClient.addRequestInterceptor(new PreemptiveAuthInterceptor(), 0);
(...)
static class PreemptiveAuthInterceptor implements HttpRequestInterceptor {
public void process(final HttpRequest request, final HttpContext context) throws HttpException, IOException {
AuthState authState = (AuthState) context.getAttribute(HttpClientContext.TARGET_AUTH_STATE);
// If no auth scheme available yet, try to initialize it
// preemptively
if (authState.getAuthScheme() == null) {
CredentialsProvider credsProvider = (CredentialsProvider) context.getAttribute(HttpClientContext.CREDS_PROVIDER);
HttpHost targetHost = (HttpHost) context.getAttribute(HttpCoreContext.HTTP_TARGET_HOST);
Credentials creds = credsProvider.getCredentials(new AuthScope(targetHost.getHostName(), targetHost.getPort()));
if (creds == null) {
throw new HttpException("No credentials for preemptive authentication");
}
authState.update(new BasicScheme(), creds);
}
}
}
Of course, you have to set the credentials provider:
httpClient.getCredentialsProvider().setCredentials(
new AuthScope(url.getHost(), url.getPort()),
new UsernamePasswordCredentials(username, password))
The AuthScope must not contain realm, as it is not known in advance.

A lot of the answers above use deprecated code. I am using Apache SOLRJ version 5.0.0.
My code consists of
private HttpSolrClient solrClient;
private void initialiseSOLRClient() {
URL solrURL = null;
try {
solrURL = new URL(urlString);
} catch (MalformedURLException e) {
LOG.error("Cannot parse the SOLR URL!!" + urlString);
throw new SystemException("Cannot parse the SOLR URL!! " + urlString, e);
}
String host = solrURL.getHost();
int port = solrURL.getPort();
AuthScope authScope = new AuthScope(host, port);
BasicTextEncryptor textEncryptor = new BasicTextEncryptor();
textEncryptor.setPassword("red bananas in the spring");
String decryptPass = textEncryptor.decrypt(pass);
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(userName, decryptPass);
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(
authScope,
creds);
HttpClientBuilder builder = HttpClientBuilder.create();
builder.addInterceptorFirst(new PreemptiveAuthInterceptor());
builder.setDefaultCredentialsProvider(credsProvider);
CloseableHttpClient httpClient = builder.build();
solrClient = new HttpSolrClient(urlString, httpClient);
}
The PreemptiveAuthInterceptor is now as follows:-
static class PreemptiveAuthInterceptor implements HttpRequestInterceptor {
public void process(final HttpRequest request, final HttpContext context) throws HttpException, IOException {
AuthState authState = (AuthState) context.getAttribute(HttpClientContext.TARGET_AUTH_STATE);
// If no auth scheme available yet, try to initialize it
// preemptively
if (authState.getAuthScheme() == null) {
CredentialsProvider credsProvider = (CredentialsProvider)
context.getAttribute(HttpClientContext.CREDS_PROVIDER);
HttpHost targetHost = (HttpHost) context.getAttribute(HttpCoreContext.HTTP_TARGET_HOST);
AuthScope authScope = new AuthScope(targetHost.getHostName(), targetHost.getPort());
Credentials creds = credsProvider.getCredentials(authScope);
if(creds == null){
}
authState.update(new BasicScheme(), creds);
}
}
}

A little late to the party but I came accross the thread trying to solve this for proxy pre-authorization of a post request. To add to Adam's response, I found the following worked for me:
HttpPost httppost = new HttpPost(url);
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(username, password);
Header bs = new BasicScheme().authenticate(creds, httppost);
httppost.addHeader("Proxy-Authorization", bs.getValue());
Thought that might be helpful for anyone else who runs into this.

I think the best way may be to just do it manually. I added the following function
Classic Java:
import javax.xml.bind.DatatypeConverter;
...
private static void addAuthHeader(HttpRequestBase http, String username, String password) throws UnsupportedEncodingException {
String encoded = DatatypeConverter.printBase64Binary((username + ":" + password).getBytes("UTF-8"));
http.addHeader("AUTHORIZATION", "Basic " + encoded);
}
HTTPRequestBase can be an instance of HttpGet or HttpPost
Android:
import android.util.Base64;
...
private static void addAuthHeader(HttpRequestBase http, String username, String password) throws UnsupportedEncodingException {
String encoded = Base64.encodeToString((username + ":" + password).getBytes("UTF-8"), Base64.NO_WRAP);
http.addHeader("AUTHORIZATION", "Basic " + encoded);
}

I'm using this code, based on my reading of the HTTPClient 4.5 docs:
HttpClientContext ctx = HttpClientContext.create()
ctx.setCredentialsProvider(new BasicCredentialsProvider())
ctx.setAuthCache(new BasicAuthCache())
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(user, pass)
AuthScope authScope = new AuthScope(host, port)
ctx.getCredentialsProvider.setCredentials(authScope, credentials)
// This part makes authentication preemptive:
HttpHost targetHost = new HttpHost(host, port, scheme)
ctx.getAuthCache.put(targetHost, new BasicScheme())
...and make sure you always pass that context to HTTPClient.execute().

I don't quite get your closing comment. It's the HttpClient that has all of that machinery for doing preemptive auth, and you only have to do that once (when you construct and configure your HttpClient). Once you've done that, you construct your method instances the same way as always. You don't "add the BasicHttpContext" to the method.
Your best bet, I'd think, is to have your own object that sets up all of the junk required for preemptive auth, and has a simple method or methods for executing requests on given HTTPMethod objects.

in android,Mat Mannion's answer can't resolve https,still send two requests,you can do like below,the trick is append authHeader with user-agent：
public static DefaultHttpClient createProxyHttpClient() {
try {
final DefaultHttpClient client = createPlaintHttpClient();
client.setRoutePlanner(new HttpRoutePlanner() {
#Override
public HttpRoute determineRoute(HttpHost target, HttpRequest request, HttpContext context) throws HttpException {
boolean isSecure = "https".equalsIgnoreCase(target.getSchemeName());
if (needProxy) {
Header header = isSecure ? ProxyUtils.createHttpsAuthHeader() : ProxyUtils.createAuthHeader();
if (isSecure) {
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, com.netease.cloudmusic.utils.HttpRequest.USER_AGENT + "\r\n" + header.getName() + ":" + header.getValue());
} else {
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, com.netease.cloudmusic.utils.HttpRequest.USER_AGENT);
if (request instanceof RequestWrapper) {
request = ((RequestWrapper) request).getOriginal();
}
request.setHeader(header);
}
String host = isSecure ? ProxyUtils.SECURE_HOST : ProxyUtils.HOST;
int port = isSecure ? ProxyUtils.SECURE_PORT : ProxyUtils.PORT;
return new HttpRoute(target, null, new HttpHost(host, port), isSecure);
} else {
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, com.netease.cloudmusic.utils.HttpRequest.USER_AGENT);
return new HttpRoute(target, null, isSecure);
}
}
});
return client;
} catch (Exception e) {
e.printStackTrace();
return new DefaultHttpClient();
}
}
public static DefaultHttpClient createPlaintHttpClient() {
try {
KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
trustStore.load(null, null);
PlainSSLSocketFactory socketFactory = new PlainSSLSocketFactory(trustStore);
socketFactory.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
BasicHttpParams params = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(params, 30000);
HttpConnectionParams.setSoTimeout(params, 30000);
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
HttpProtocolParams.setContentCharset(params, HTTP.UTF_8);
SchemeRegistry registry = new SchemeRegistry();
registry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80));
registry.register(new Scheme("https", socketFactory, 443));
ThreadSafeClientConnManager ccm = new ThreadSafeClientConnManager(params, registry);
HttpClientParams.setCookiePolicy(params, CookiePolicy.BROWSER_COMPATIBILITY);
final DefaultHttpClient client = new DefaultHttpClient(ccm, params);
client.setRoutePlanner(new HttpRoutePlanner() {
#Override
public HttpRoute determineRoute(HttpHost target, HttpRequest arg1, HttpContext arg2) throws HttpException {
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, com.netease.cloudmusic.utils.HttpRequest.USER_AGENT);
return new HttpRoute(target, null, "https".equalsIgnoreCase(target.getSchemeName()));
}
});
return client;
} catch (Exception e) {
e.printStackTrace();
return new DefaultHttpClient();
}
}

SolrConfig:
#Configuration
public class SolrConfig {
#Value("${solr.http.url}")
private String solrUrl;
#Value("${solr.http.username}")
private String solrUser;
#Value("${solr.http.password}")
private String solrPassword;
#Value("${solr.http.pool.maxTotal}")
private int poolMaxTotal;
#Value("${solr.http.pool.maxPerRoute}")
private int pollMaxPerRoute;
#Bean
public SolrClient solrClient() {
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(poolMaxTotal);
connectionManager.setDefaultMaxPerRoute(pollMaxPerRoute);
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(solrUser, solrPassword));
CloseableHttpClient httpClient = HttpClientBuilder.create()
.addInterceptorFirst(new PreemptiveAuthInterceptor())
.setConnectionManager(connectionManager)
.setDefaultCredentialsProvider(credentialsProvider)
.build();
return new HttpSolrClient.Builder(solrUrl).withHttpClient(httpClient).build();
}
}
PreemptiveAuthInterceptor:
public class PreemptiveAuthInterceptor implements HttpRequestInterceptor {
public void process(final HttpRequest request, final HttpContext context)
throws HttpException {
AuthState authState = (AuthState) context
.getAttribute(HttpClientContext.TARGET_AUTH_STATE);
// If no auth scheme available yet, try to initialize it
// preemptively
if (authState.getAuthScheme() == null) {
CredentialsProvider credentialsProvider = (CredentialsProvider) context
.getAttribute(HttpClientContext.CREDS_PROVIDER);
HttpHost targetHost = (HttpHost) context
.getAttribute(HttpCoreContext.HTTP_TARGET_HOST);
Credentials credentials = credentialsProvider.getCredentials(new AuthScope(
targetHost.getHostName(), targetHost.getPort()));
if (credentials == null) {
throw new HttpException(
"No credentials for preemptive authentication");
}
authState.update(new BasicScheme(), credentials);
}
}
}