We Keep Coding

Assign a different proxy to thread - Java

Hi Guys I am making a bot which can use a an other REST API to create a user but API only supports one user at a time and I dont want to change it. So I am using Multithreading to call the API multiple times, but I want to use proxies in it. So like different proxies for different threads, and all the threads use HttpClient and I tried its #proxy method and gave it the ip and port of the proxy but when I tried to call the API it returned a null response and I tried without the proxy and it did return a valid response.
So is there any other way to assign an proxy to a thread
My Code
HttpClient client = HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_1_1)
.followRedirects(HttpClient.Redirect.NORMAL)
//THIS LINE BELOW IS HOW I USED PROXIES
.proxy(ProxySelector.of(new InetSocketAddress("proxy.example.com", 80)))
.connectTimeout(Duration.ofSeconds(30))
.build();
HttpRequest discordAccNoCaptcha = HttpRequest.newBuilder()
.uri(URI.create("https://myapiserver.com/api/v9/auth/register"))
.timeout(Duration.ofMinutes(2))
.POST(HttpRequest.BodyPublishers.ofString(body))
.build();
String response = client.send(discordAccNoCaptcha, HttpResponse.BodyHandlers.ofString()).body(); /* This is null when using ProxySelector and valid response when using no ProxySelector */

How to set and handle timeout in Spring WebClient?

Spring docs says it is required to configure http client for WebClient manually to set timeouts: https://docs.spring.io/spring/docs/current/spring-framework-reference/web-reactive.html#webflux-client-builder-reactor-timeout.
But since WebClient returns reactive Mono, it's possible (api-wise) to apply .timeout method.
Does it have the same effect?
Moreover, when one uses .timeout method, Reactor's TimeoutException is expected. Will the same error appear in the stream if configuration is done manually i.e. will doOnError(TimeoutException.class, ...) work?

My findings
Setting a timeout in a http client specific way will lead to http client specific exception i.e. WebClient doesn't wrap exceptions:
#Test
void test() {
var host = "localhost";
var endpoint = "/test";
var port = 8089;
var timeout = Duration.ofSeconds(3);
WireMockServer wireMockServer = new WireMockServer(wireMockConfig().port(8089));
wireMockServer.start();
WireMock.configureFor(host, wireMockServer.port());
WireMock.stubFor(get(urlEqualTo(endpoint))
.willReturn(aResponse().withFixedDelay((int) timeout.toMillis())));
HttpClient httpClient = HttpClient.create()
.tcpConfiguration(client ->
client.doOnConnected(conn -> conn
.addHandlerLast(new ReadTimeoutHandler((int) (timeout.toSeconds() / 2)))
.addHandlerLast(new WriteTimeoutHandler((int) (timeout.toSeconds() / 2)))));
WebClient webClient = WebClient.builder()
.baseUrl(format("http://%s:%d", host, port))
.clientConnector(new ReactorClientHttpConnector(httpClient)).build();
webClient.get().uri(endpoint).retrieve().bodyToMono(Recommendation.class).block();
}
This will lead to io.netty.handler.timeout.ReadTimeoutException.
.timeout(timeout.dividedBy(2)).block() leads to regular TimeoutException (java.util.concurrent) but it's still an open question whether a web client takes care about connections afterwards (probably not).
My solution is to use http client specific configuration to ensure native and correct way to utilize connections while adding new handler that wraps http client related exception into more generic ones (or java.util.concurrent.TimeoutException) so that WebClient clients won't depend on provider exceptions.

Configuring Apache HttpClient to access service through proxy/load-balancer (overriding Host header)

I am having a problem getting the Apache HttpClient to connect to a service external to my virtualised development environment.
To access the internet (e.g. api.twitter.com) I need to call a local URL (e.g. api.twitter.com.dev.mycompany.net), which then forwards the request to real host.
The problem is, that to whatever request I send, I get a 404 Not Found response.
I have tried debugging it using wget, and it appears the problem is, that the destination server identifies the desired resource by using both the request URL and the hostname in the Host header. Since the hostname does not match, it is unable to locate the resource.
I have (unsuccessfully) tried to override the Host header by setting the http.virtual-host parameter on the client like this:
HttpClient client = new DefaultHttpClient();
if (envType.isWithProxy()) {
client.getParams().setParameter(ClientPNames.VIRTUAL_HOST, "api.twitter.com");
}
Technical details:
Client is used as an executor in RESTeasy to call the REST API. So "manually" setting the virtual host (as described here) is not an option.
Everything is done via HTTPS/SSL - not that I think it makes a difference.
Edit 1: Using a HttpHost instead of a String does not have the desired effect either:
HttpClient client = new DefaultHttpClient();
if (envType.isWithProxy()) {
HttpHost realHost = new HttpHost("api.twitter.com", port, scheme);
client.getParams().setParameter(ClientPNames.VIRTUAL_HOST, realHost);
}
Edit 2: Further investigation has revealed, that the parameter needs to be set on the request object. The following is the code v. 4.2-aplha1 of HttpClient setting the virtual host:
HttpRequest orig = request;
RequestWrapper origWrapper = wrapRequest(orig);
origWrapper.setParams(params);
HttpRoute origRoute = determineRoute(target, origWrapper, context);
virtualHost = (HttpHost) orig.getParams().getParameter(
ClientPNames.VIRTUAL_HOST);
paramsare the parameters passed from the client. But the value for 'virtualHost' is read from the request parameters.
So this changes the nature of the question to: How do I set the VIRTUAL_HOST property on the requests?

ClientPNames.VIRTUAL_HOST is the right parameter for overriding physical host name in HTTP requests. I would just recommend setting this parameter on the request object instead of the client object. If that does not produce the desired effect please post the complete wire / context log of the session (see logging guide for instructions) either here or to the HttpClient user list.
Follow-up
OK. Let's take a larger sledge hammer. One can override content of the Host header using an interceptor.
DefaultHttpClient client = new DefaultHttpClient();
client.addRequestInterceptor(new HttpRequestInterceptor() {
public void process(
final HttpRequest request,
final HttpContext context) throws HttpException, IOException {
request.setHeader(HTTP.TARGET_HOST, "www.whatever.com");
}
});
One can make the interceptor clever enough to override the header selectively, only for specific hosts.

In a JAX-WS web service, how do I force a socket close after each Soap call is returned?

I am using com.sun.httpserver.HttpServer and javax.xml.ws.Endpoint to publish a JAX-WS web service, which was generated by running wsimport on an existing WSDL and implementing the genereated service interface. All this was part of JDK 1.6 (JAX-WS RI 2.1.6). My web service, running as a Java program without an additional Web container is supposed to simulate an existing SOAP service that was implemented using Apache Axis, running on Tomcat. Existing clients are likewise implemented using Apache Axis.
The problem I am having is that Soap operation calls from the clients to my JAX-WS service hang for a while, and then end with socket time-out on the client's side. This occurs even though the JAX-WS service is returning the SOAP response right away.
Inspecting the packets with tcpdump and wireshark, I noticed that with the existing Axis web service, after the SOAP response is sent from the server to the client, the server sends a "FIN ACK" packet, to which the clients responds with "FIN ACK". This concludes all packets pertinent to the SOAP operation. On the other hand, when running with the JAX-WS service, the server does not send a "FIN ACK" after the SOAP response is sent to the client. And the client seems to continue reading the socket input stream for it.
This leads me to believe that the JAX-WS web service stack is somehow keeping the socket open, even after the response to a SOAP call has been sent. And it appears that the client is expecting the socket to close at the end of a SOAP operation. Unfortunately, I cannot modify the client to behave differently.
Is there a way to configure the Endpoint or the HttpServer instances that I am using to publish the JAX-WS service to always close a socket after each SOAP operation?
I tried setting the system property http.keepAlive to false, but this did not seem to make any difference.
Thanks in advance for your help.

I found a way around this problem, but it's not very elegant. Essentially I get the HttpHandler object from the HttpContext after it's been created by the Endpoint.publish operation. And I call its handle() method from another HttpHandler class I wrote, which follows it up by sending a HttpHeader with "Connection" set to "close". Something like this:
...
HttpServer server = HttpServer.create(myInetSocketAddress, 5);
HttpContext context = server.createContext(mySoapPath);
Endpoint endpoint = Endpoint.create(mySoapImpl);
endpoint.publish(context);
MyHandler handler = new MyHandler(context.getHandler());
server.removeContext(mySoapPath);
server.createContext(mySoapPath, handler);
server.start();
...
private class MyHandler implements HttpHandler {
private HttpHandler h;
public MyHandler(HttpHandler in) {
h = in;
}
public void handle(HttpExchange t) throws IOException {
h.handle(t);
t.getResponseHeaders().set("Connection", "close");
t.sendResponseHeaders(200, 0);
t.close();
}
}
It works for my current needs, but it just seems like there's got to be a better way. Please do post if you have another solution. Thanks!

When I had this problem in Java 8 all requests from JAX-WS RI 2.2.9 had header: Connection: keep-alive
To force closing TCP connection after each request I had to change this header to Connection: close
I had done it by:
SoapService service = new SoapService(url);
SoapPort port = service.getSomePort();
Map<String, List<String>> requestHeaders = new HashMap<>();
requestHeaders.put("Connection", Collections.singletonList("close"));
BindingProvider bindingProvider = (BindingProvider)port;
bindingProvider.getRequestContext().put(MessageContext.HTTP_REQUEST_HEADERS, requestHeaders);
where:
public static final String HTTP_REQUEST_HEADERS = "javax.xml.ws.http.request.headers";

ProxySelector changes URL's scheme from https:// to socket://

I need to access Facebook but all outgoing communication is blocked on our server so I have to use proxy.
I initialize proxies with:
ProxySelector.setDefault(new ConfigurableProxySelector(mapping));
Proxy type is HTTP, proxy host and port are working (confirmed by simple wget test).
I'm trying to do this:
HttpClient httpClient = new HttpClient();
HttpMethod method = new GetMethod("https://graph.facebook.com:443");
int status = httpClient.executeMethod(method);
Now, in my class ConfigurableProxySelector I have select method on which I have breakpoint:
public List<Proxy> select(URI uri) {
...
}
So, using HttpClient I make an request, which should be proxied and code stops at breakpoint in select() method in ConfigurableProxySelector.
But what is strange is that uri.scheme = "socket" and .toString() gives "socket://graph.facebook.com:443" instead of "https://graph.facebook.com:443".
Because ProxySelector have mapping for "https://" and not for "socket://", it does not find it and it ends with "Connection refused". What is strange is that select() method is called 4 times before execution ends with "Connection refused".
Any help would be appreciated.

Apache HTTP Client 3.1 will not natively honor HTTP Proxies returned from the default ProxySelector or user implementations.
Quick Summary of ProxySelector
ProxySelector is a service class which selects and returns a suitable Proxy for a given URL based on its scheme. For example, a request for http://somehost will try to provide an HTTP proxy if one is defined. The default ProxySelector can be configured at runtime using System Properties, such as http.proxyHost and http.proxyPort.
HTTPUrlConnection
An instance of HTTPUrlConnection will check against the default ProxySelector multiple times: 1st to select for http or https, then later when it builds the raw tcp socket, using the socket scheme. A SOCKS proxy could be used to proxy a raw tcp socket but are not often found in corporate environments, so a raw tcp socket will usually receive no proxy.
HTTP Client 3.1
HC 3.1, on the other hand, will never check the default ProxySelector for the http/https schemes. It will check, however, at a later points for the socket scheme when it eventually builds the raw socket - This is the request you are seeing. This means the System Properties http.proxyHost and http.proxyPort are ineffective. This is obviously not ideal for most people who only have an HTTP/HTTPS proxy.
To work around this, you have two options: define a proxy on each HC 3.1 connection or implement your own HC 3.1 HTTPConnectionManager.
HTTPConnectionManager
The HTTPConnectionManager is responsible for building connections for the HC 3.1 client.
The default HC 3.1 HTTPConnectionManager can be extended so that it looks for a suitable proxy from a ProxySelector (default or custom) when building the request in the same way HTTPUrlConnection does:
public class MyHTTPConnectionManager extends SimpleHttpConnectionManager {
#Override
public HttpConnection getConnectionWithTimeout(
HostConfiguration hostConfiguration, long timeout) {
HttpConnection hc = super.getConnectionWithTimeout(hostConfiguration, timeout);
try {
URI uri = new URI( hostConfiguration.getHostURL());
List<Proxy> hostProxies = ProxySelector.getDefault().select(uri);
Proxy Proxy = hostProxies.get(0);
InetSocketAddress sa = (InetSocketAddress) Proxy.address();
hc.setProxyHost(sa.getHostName());
hc.setProxyPort(sa.getPort());
} catch (URISyntaxException e) {
return hc;
}
return hc;
}
}
Then, when you create an HC 3.1 client, use your new connection manager:
HttpClient client = new HttpClient(new MyHTTPConnectionManager() );

It's not the ProxySelector that changes the scheme, but the SocketFactory opening a Socket.
If the SocketFactory is null a SOCKS socket will be created by default which only allows SOCKS proxies. I don't know anything about Sockets and cannot tell you if there's a way to make it work with HTTP proxies.
But using another approach may help, since Apache HttpClient seems to have its own way to configure proxies.
client.getHostConfiguration().setProxy(proxyHost, proxyPort);
if (proxyUser != null) {
client.getState().setProxyCredentials(new AuthScope(proxyHost, proxyPort),
new UsernamePasswordCredentials(proxyUser, proxyPassword));
}