We Keep Coding

Initializing JestClient when application calls multiple Elasticsearch endpoints

My API currently calls one Elasticsearch endpoint using JestClient. I want to add some functionality that requires calling a second, different Elasticsearch endpoint. How is this possible, when you have to specify the endpoint upon initializing JestClient?
#Provides
#Singleton
public JestClient jestClient() {
JestClientFactory factory = new JestClientFactory();
factory.setHttpClientConfig(
new HttpClientConfig.Builder("http://localhost:9200")
.build());
return factory.getObject();
}
My application design uses Singleton classes for these initializations so I'm not sure how to fix this aside from using a different Elasticsearch client for my second endpoint.

Or you can create two clients and based on the use-case in calling classes, if you require both the clients, inject both clients or if you requires one specific client, inject that client in your class, you just need to have a different name(like clientv2 for es 2 version, and clientv5 for ES 5.x version) for your client to make it work, its easy as well, as you know your use-case and know what all versions of clients is need in your classes.
on a side note, active development of JEST is stopped long ago, and now elasticsearch provides the official java client know as Java high level rest client, so IMHO you should switch to JHLRC to get the maximum benefit and to make future migration easy, which I think is the use-case of yours.

You could create a factory and return the client depending on a parameter.
The parameter could be the url itself or just an enum that represents the endpoint.

How to configure and build a custom RestTemplate for each client in Spring?

I am using Spring RestTemplate for executing HTTP request from my application. There are several services we need to call, some on the internet and some on intranet, some fast and some slow. I have been instructed to configure custom settings for each service, basically, connection timeout, read timeout.
These setting will be extremely specific for example, the services hosted on intranet would have a timeout of ~2-5s while they provide an SLA of 1000ms for 99.9% of requests. While the other third party services with ~10-60s.
As these parameters can be set in the factory for the template only, i am creating a number of beans with different factories differing in timeouts only. Something like this:
#Bean
RestTemplate restTemplate() {
HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory(httpClient);
factory.setReadTimeout(20000);
factory.setConnectTimeout(5000);
RestTemplate restTemplate = new RestTemplate(factory);
}
I am afraid this will create a maintenance nightmare eventually. Can it be solved in a better ways?
PS: The application is a monolith calling various services.

You will have to create multiple RestTemplates and assign timeouts, connection pool size. Connection pool will improve the performance drastically
I have hard-coded the connection properties, you can pick it from application.properties file
#Configuration
class RestTemplateConfigs {
#Bean
public HttpClient httpClient() {
return HttpClientBuilder.create()
.setMaxConnPerRoute(200)
.setMaxConnTotal(50)
.setConnectionTimeToLive(10L, TimeUnit.SECONDS)
.build();
}
#Bean(name = "restTemplate1")
RestTemplate restTemplate() {
HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory(httpClient());
RestTemplate restTemplate = new RestTemplate(factory);
return restTemplate;
}
}
You can create multiple RestTemplates and Autowire it using Qualifier name.

Disclaimer: my answer suggests to work with another Http client than Rest Template - If you must use Rest template my answer would be irrelevant.
I dealt with a similar design issue and here is what I did. I wrote my own HttpClient class. It is much simpler in use then most known Http clients. This class could be used on its own or (and this is relevant for your case) it could be used as a parent class for a group of classes (implementing the same interface) where each class will be an Http client for specific Rest Service. On this class you can pre-set the target URL and all the parameters (such as read and connection timeouts etc). Once this class is preset, all you need to do is to invoke sendHttpRequestMethod(). Just to expand a bit - lets say you have a User Rest service with CRUD API implemented by particular URL calls with different HTTP methods and may be different URLs. (say in addition to create (POST) update (PUT) read (GET) and delete (DELETE) methods that are located at HTTP://www.myserver.com:8083/user say you will also have methods activate and deactivate (say both GET) at URLs HTTP://www.myserver.com:8083/user/activate/ and HTTP://www.myserver.com:8083/user/deactivate.
So, in this case, your Http client will set all required timeouts and other configurations and it will also have pre-set target URL HTTP://www.myserver.com:8083/user. and it will have six methods as mentioned above, where each one will simply invoke the parent class method sendHttpRequest(). Of course, for activate and deactivate methods you will need to append "activate" and "deactivate" suffixes for pre-set base URL. So, for each REST service, you can create a dedicated Http client with very minimal effort since the base class already does most of the job. In addition to it, I wrote a self-populating factory for any group of classes that implement the same interface. With that factory, all you will have to do is to write your additional Http client and the factory will detect it and will make it available on its own by predefined name or by the name of the class (based on your choice). All this worked so well for me, that I packaged it into Open Source library called MgntUtils and published it on Maven and Github (with source code and Javadoc. JavaDoc is available here). A detailed explanation on the Self-populating factory can be seen in Javadoc here. Also, the general article about the library can be found here and specific article about the idea and the implementation of self-populating Factory can be found here. Package com.mgnt.lifecycle.management.example in the source code contains a working example. I hope this helps.

Using parameterized construction of the RestTemplate bean solved my problem.
The bean is now configured as:
#Bean
#Scope(BeanDefinition.SCOPE_PROTOTYPE)
public RestTemplate getRestTemplate(String configParam){
int connectionTimeout; //get from config using configParam
int readTimeout; //get from config using configParam
HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory();
factory.setConnectTimeout(connectionTimeout);
factory.setReadTimeout(readTimeout);
return new RestTemplate(factory);
}
Instead of #Autowiring this field, it can be injected in may be #PostConstruct method like depicted below. The dependent bean can do following:
#Autowire
BeanFactory beanFactory;
RestTemplate restTemplate;
#PostConstruct
public void init(){
restTemplate = beanFactory.getBean(RestTemplate.class, configParam);
}
Here you can have your bean with custom settings injected in restTemplate.

Okhttp 3.x dynamically add/remove authenticator / interceptor

Is it possible to add/remove Authenticators and/or Interceptors to an existing Okhttp instance? If yes, how?

No, it is not possible.
However, you can create a builder from an existing client, and make changes to that. This will share the dispatcher, connectionPool etc.
OkHttpClient.Builder clientBuilder = client1.newBuilder();
clientBuilder.networkInterceptors().add(0, serviceInterceptor);
OkHttpClient client2 = clientBuilder.build();
There is an example for adjusting the timeout of a client in the javadoc https://square.github.io/okhttp/3.x/okhttp/okhttp3/OkHttpClient.html

Is there something like DefaultMqttPahoClientFactory for creating an async client?

I would like to use an async Paho client in Spring Integration:
#Bean
public DefaultMqttPahoClientFactory clientFactory() {
return new DefaultMqttPahoClientFactory();
}
<int-mqtt:message-driven-channel-adapter id="mqttInbound"
client-id="${mqtt.default.client.id}.src"
url="${mqtt.url}"
topics="sometopic"
client-factory="clientFactory"
channel="output"/>
Do I have to write one on my own? I can only find examples for sync client factories.

Correct; at this time, only the sync client is supported.
Feel free to open an improvement JIRA issue here and, of course contributions are always welcome - guidelines here.
That said, it's not clear to me the benefit of using an async client in the inbound adapter as in your example; it is message-driven and arriving messages are delivered asynchronously on the client's thread. Obviously, the outbound adapter could gain some benefit from async operations.
The outbound adapter is pretty simple and shouldn't be too hard to write one that uses the async client - it would probably need to send the delivery tokens to some channel - but we should abstract away from the IMqttDeliveryToken object so we don't leak Paho into the upper layers.

I'm pretty sure that the DefaultMqttPahoClientFactory() provided by Spring Integration is using the "synchronous" mode of operation in Paho, so the likelihood is that you would need an alternative ClientFactory implementation.

Setting proxies in HtmlUnit

I'm using proxies with htmlunit, my proxy list contain mixture of both http and socks, I dont know if the next selected proxy to be passed to htmlunit is type http or socks, will htmlunit automatically determine the type and use the appropriate rule for connecting through that proxy?

I've made an application which used mixture of proxies as well, but it was a while ago. In that version of HtmlUnit, it required being explicitly told if the proxy was SOCKS or not, otherwise it assumed it was a HTTP proxy. I looked briefly in the change logs, and found nothing indicating that this had changed, so the answer should be no, it will assume that the proxy is HTTP if you don't tell HtmlUnit that the proxy is SOCKS.
To check what type a proxy is, one can use something like:
SocketAddress addr = new InetSocketAddress("proxyAddress", port);
Proxy proxy = new Proxy(Proxy.Type.HTTP, addr); //or Proxy.Type.SOCKS
URL url = new URL("http://google.com");
URConnection conn = url.openConnection(proxy);
If the code fails (i.e throws an exception), then the proxy is most likely either dead or SOCKS. (HtmlUnit will throw an exception in the first case anyway, or you can perform the same test again with Proxy.Type.SOCKS if you aren't certain the proxy is alive.)

Check out Java Networking and Proxies.
It talks about multiple strategies for setting proxies. It also gives option to provide multipe proxy to same connection through proxy selector.

The other two answers are about how to generically use a proxy in a Java program, but it is a little bit different with HtmlUnit. Setting the proxy on the Java process does nothing; instead you want the simulated browser to use the proxy.
int myProxyPort=8080;
WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER_8, "myproxyhost.com", myProxyPort);

This framework detects the proxy type for you and can instantiate a HtmlUnit WebClient with correct Socks/Http/Https proxy configuration for you: https://github.com/subes/invesdwin-webproxy
Essentially it automates the trial&error approach when maintaining its proxy list for you. So you don't have to worry about that in your own code.