We Keep Coding

Connecting to ES with Spring Data Elasticsearch (reactive) gives error host not reachable

I'm running on an aws-elasticsearch (with OpenSearch 1.1.x) service and im trying to connect with it from a spring application using spring-data-elasticsearch, according to the doc i configured the bean as it says.
on my local i used a ssh tunnel from my aws account.
i used this command:
ssh -4 -i my-creds.pem ec2-user#xxxx.xxxx.xxxx.xxxx -N -L 9200:vpc-my-custom-domain-etc.us-east-1.es.amazonaws.com:443
so i can connect with OpenSearch dashboard over localhost in my browser through port 9200.
Using the OpenSearch RestHighLevelClient from OpenSearch and disabling the ssl i can connect and it works just fine here the config with OS RHLC:
import org.apache.http.HttpHost;
import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.client.indices.CreateIndexRequest;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.Map;
public class OSSCLientWorks{
private static final Logger log = LoggerFactory.getLogger(ClientAutoWrapper.class);
public void request(String indexName, Map<String, Object> doc) throws IOException {
//Create a client.
RestClientBuilder builder = RestClient.builder(new HttpHost("localhost", 9200, "https"))
.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder
//.addInterceptorFirst(interceptor) //-> for AwsRequestInterceptor due to some struggles i had, not necessary to work with localhost
.setSSLHostnameVerifier((hostname, session) -> true));
try (RestHighLevelClient hlClient = new RestHighLevelClient(builder)) {
CreateIndexRequest createIndexRequest = new CreateIndexRequest(indexName);
var createIndexResp = hlClient.indices().create(createIndexRequest, RequestOptions.DEFAULT);
log.info("Create index resp {}", createIndexResp);
IndexRequest indexRequest = new IndexRequest(createIndexResp.index())
.id(String.valueOf(doc.get("id")))
.source(doc);
var response = hlClient.index(indexRequest, RequestOptions.DEFAULT);
var resp = response.toString();
log.info("response is {}", json);
}
}
}
, but when i try with spring and its reactive client i get this error:
reactor.core.Exceptions$ErrorCallbackNotImplemented: org.springframework.data.elasticsearch.client.NoReachableHostException: Host 'localhost:9200' not reachable. Cluster state is offline.
Caused by: org.springframework.data.elasticsearch.client.NoReachableHostException: Host 'localhost:9200' not reachable. Cluster state is offline.
here is the config i used to work with spring-data-elasticsearch:
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.client.ClientConfiguration;
import org.springframework.data.elasticsearch.client.reactive.ReactiveElasticsearchClient;
import org.springframework.data.elasticsearch.client.reactive.ReactiveRestClients;
import org.springframework.data.elasticsearch.config.AbstractReactiveElasticsearchConfiguration;
import org.springframework.data.elasticsearch.core.ReactiveElasticsearchOperations;
import org.springframework.data.elasticsearch.core.ReactiveElasticsearchTemplate;
import org.springframework.data.elasticsearch.repository.config.EnableElasticsearchRepositories;
import org.springframework.data.elasticsearch.repository.config.EnableReactiveElasticsearchRepositories;
#Configuration
#EnableReactiveElasticsearchRepositories(basePackages = {"com.elastic.repo"})
public class ElasticRestHighLevelClientConfig extends AbstractReactiveElasticsearchConfiguration {
#Override
#Bean
public ReactiveElasticsearchClient reactiveElasticsearchClient() {
final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo("localhost:9200")
.build();
return ReactiveRestClients.create(clientConfiguration);
}
#Bean
public ReactiveElasticsearchOperations elasticsearchOperations(ReactiveElasticsearchClient reactiveElasticsearchClient) {
return new ReactiveElasticsearchTemplate(reactiveElasticsearchClient);
}
}
i also tried some solutions other people posted here on SO and Github, but the problem persists, does anybody have a workaround for this? what am i doing wrong?
here i did a demo for the trouble
Thank you very much in advance!
EDIT: clarity

You have to configure to use SSL for the reactive client with one of the usingSsl()methods:
#Override
#Bean
public ReactiveElasticsearchClient reactiveElasticsearchClient() {
final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo("localhost:9200")
.usingSsl() // <--
.build();
return ReactiveRestClients.create(clientConfiguration);
}

NoReachableHostException is just a generic error they throw when lookupActiveHost(HostProvider interface) fails.
You should debug what happens before - for you it's probably here:
#Override
public Mono clusterInfo() {
return createWebClient(endpoint) //
.head().uri("/") //
.exchangeToMono(it -> {
if (it.statusCode().isError()) {
state = ElasticsearchHost.offline(endpoint);
} else {
state = ElasticsearchHost.online(endpoint);
}
return Mono.just(state);
}).onErrorResume(throwable -> {
state = ElasticsearchHost.offline(endpoint);
clientProvider.getErrorListener().accept(throwable);
return Mono.just(state);
}).map(elasticsearchHost -> new ClusterInformation(Collections.singleton(elasticsearchHost)));
}
see what is the real exception on error resume.
I bet you will get SSL Handshake Exception, you can fix it in the clientConfiguration with .usingSsl({SSL CONTEXT HERE})
You can create insecure context like this(convert to java if needed):
SSLContext.getInstance("TLS")
.apply { init(null, InsecureTrustManagerFactory.INSTANCE.trustManagers, SecureRandom()) }

Wiremock simulate a proxy server running

I want to add a test for the following e2e scenario:
My app is making a web request to an external service through an internal proxy server, the proxy server manipulates the request body, forwards the request to the desination host and returns the response returned.
Say for example
I do a post request to external.service/an/endpoint (through my-proxy-server) the body
{
"card_number": "<proxy server pls fill the cc details>"
}
The proxy server modifies the request to fill the cc details and forwards it to external.service/an/endpoint with body
{
"card_number": "372735466563005"
}
The external.service returns status OK. proxy server returns the response without modifying.
How do I test this workflow with wiremock? I can do WireMock.stubFor() for external.service, But I don't know how to make wiremock proxy work with the proxy setting of my Webclient.
See the test, actually, Rest Template test, restTemplateWithWireMockAsProxyServer works as expected, routing my requests through the proxy, but the webClientWithWireMockAsProxyServer errors out with my RCA:
20:06:59.165 [qtp105751207-24] DEBUG wiremock.org.eclipse.jetty.server.HttpChannel - REQUEST for //localhost:58978localhost:58978 on HttpChannelOverHttp#4a71ab50{r=1,c=false,c=false/false,a=IDLE,uri=//localhost:58978localhost:58978,age=0}
CONNECT //localhost:58978localhost:58978 HTTP/1.1
Host: localhost:58978
These Calls over wiremock proxy are not possible as mentioned here. But all my urls are like http://localhost:<port>, meaning I am not making any https call.
package com.spotnana.obt.supplier.services;
import com.github.tomakehurst.wiremock.WireMockServer;
import com.github.tomakehurst.wiremock.client.WireMock;
import com.github.tomakehurst.wiremock.core.WireMockConfiguration;
import com.github.tomakehurst.wiremock.http.RequestMethod;
import com.github.tomakehurst.wiremock.matching.RequestPatternBuilder;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.nio.charset.StandardCharsets;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.HttpHeaders;
import org.apache.http.conn.HttpHostConnectException;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.http.client.reactive.ReactorResourceFactory;
import org.springframework.util.SocketUtils;
import org.springframework.web.client.RestTemplate;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.netty.http.client.HttpClient;
import reactor.netty.tcp.ProxyProvider;
#Slf4j
public class SimpleWiremockProxyServerTest {
private final String HOST = "localhost";
private final String MOCK_ENDPOINT = "/my/endpoint";
private WireMockServer targetServer;
private WireMockServer proxyServer;
private WireMock targetWireMock;
private WireMock proxyWireMock;
private String targetBaseUrl;
#Before
public void setup() {
final int targetPort = SocketUtils.findAvailableTcpPort();
this.targetServer = new WireMockServer(WireMockConfiguration.wireMockConfig().port(targetPort));
this.targetServer.start();
this.targetWireMock = new WireMock(targetPort);
this.targetWireMock.resetMappings();
this.targetBaseUrl = "http://" + HOST + ":" + targetPort;
final int proxyPort = SocketUtils.findAvailableTcpPort();
this.proxyServer =
new WireMockServer(
WireMockConfiguration.wireMockConfig().port(proxyPort).enableBrowserProxying(true));
this.proxyServer.start();
this.proxyWireMock = new WireMock(proxyPort);
this.proxyWireMock.resetMappings();
}
#After
public void tearDown() throws HttpHostConnectException {
this.targetWireMock.shutdown();
this.targetServer.stop();
try {
this.proxyWireMock.shutdown();
this.proxyServer.stop();
} catch (final Exception ex) {
log.warn("Proxy server is shutdown already");
}
}
#Test
public void restTemplateWithWireMockAsProxyServer() {
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(HOST, this.proxyServer.port()));
requestFactory.setProxy(proxy);
final var reqPatternBuilder =
RequestPatternBuilder.newRequestPattern(
RequestMethod.GET, WireMock.urlEqualTo(MOCK_ENDPOINT));
final var mappingBuilder =
WireMock.get(WireMock.urlEqualTo(reqPatternBuilder.build().getUrl()));
reqPatternBuilder
.withHeader(HttpHeaders.ACCEPT, WireMock.containing(MediaType.APPLICATION_JSON_VALUE))
.withHeader(
HttpHeaders.ACCEPT_CHARSET,
WireMock.containing(StandardCharsets.UTF_8.name().toUpperCase()));
mappingBuilder.willReturn(
WireMock.aResponse()
.withStatus(HttpStatus.OK.value())
.withBody("{ \"success\": true }")
.withHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE));
this.targetWireMock.register(mappingBuilder);
ResponseEntity<String> responseEntity =
new RestTemplate(requestFactory)
.getForEntity(this.targetBaseUrl + MOCK_ENDPOINT, String.class);
Assert.assertEquals(responseEntity.getStatusCode(), HttpStatus.OK);
System.out.println("responseEntity: " + responseEntity.getBody());
}
#Test
public void webClientWithWireMockAsProxyServer() {
var client = HttpClient.create()
.tcpConfiguration(
tcpClient ->
tcpClient.proxy(
proxy -> {
proxy
.type(ProxyProvider.Proxy.HTTP)
.host(HOST)
.port(this.proxyServer.port());
}));
var webClient = WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(client))
.build();
final var reqPatternBuilder =
RequestPatternBuilder.newRequestPattern(
RequestMethod.GET, WireMock.urlEqualTo(MOCK_ENDPOINT));
final var mappingBuilder =
WireMock.get(WireMock.urlEqualTo(reqPatternBuilder.build().getUrl()));
reqPatternBuilder
.withHeader(HttpHeaders.ACCEPT, WireMock.containing(MediaType.APPLICATION_JSON_VALUE))
.withHeader(
HttpHeaders.ACCEPT_CHARSET,
WireMock.containing(StandardCharsets.UTF_8.name().toUpperCase()));
mappingBuilder.willReturn(
WireMock.aResponse()
.withStatus(HttpStatus.OK.value())
.withBody("{ \"success\": true }")
.withHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE));
this.targetWireMock.register(mappingBuilder);
var response =
webClient.get().uri(this.targetBaseUrl + MOCK_ENDPOINT).exchange().block().bodyToMono(String.class);
response.subscribe(x -> System.out.println("x:" + x));
}
}
I complains with error java.net.UnknownHostException: <proxy server>: nodename nor servname provided, or not known. Is there a way to mock the wiremock proxy server, rather than running an actual server for this. I also want to put validations in the proxy server for request-responses.

Wiremock doesn't support HTTP CONNECT method. You can try Hoverfly as an alternative to Wiremock. There is a github issue if you're interested in details.

vertx get to external address

Hi I'm trying to make a request to external service with the use of httpClient vert.x but I keep getting error: Search domain query failed. Original hostname: 'google.com' failed to resolve 'google.com'
What am i missing in the code below? I'm not sure about those AddressResolverOptions, I have read about them but still I'm not sure what they are responsible for. I'd like to make the request the simplest possible way but the more I investigate the more difficult it seems
package io.vertx.starter;
import io.vertx.core.Vertx;
import io.vertx.core.VertxOptions;
import io.vertx.core.dns.AddressResolverOptions;
import io.vertx.core.http.HttpClientOptions;
import io.vertx.ext.unit.Async;
import io.vertx.ext.unit.TestContext;
import io.vertx.ext.unit.junit.VertxUnitRunner;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import java.net.URL;
import java.util.Collections;
#RunWith(VertxUnitRunner.class)
public class MainVerticleTest {
private Vertx vertx;
#Before
public void setUp(TestContext tc) {
vertx = Vertx.vertx(new VertxOptions().setAddressResolverOptions(
new AddressResolverOptions().addSearchDomain("google.com").addSearchDomain("bar.com"))
);
// vertx = Vertx.vertx();
vertx.deployVerticle(MainVerticle.class.getName(), tc.asyncAssertSuccess());
}
#After
public void tearDown(TestContext tc) {
vertx.close(tc.asyncAssertSuccess());
}
#Test
public void testThatTheServerIsStarted(TestContext tc) {
final HttpClientOptions httpClientOptions = new HttpClientOptions();
httpClientOptions.setConnectTimeout(300);
httpClientOptions.setIdleTimeout(5);
// httpClientOptions.` `
httpClientOptions.setSsl(true).setTrustAll(true);
// final URL url = new URL("currentUrl");
Async async = tc.async();
vertx.createHttpClient(httpClientOptions).getNow(
443,"google.com", "/", response -> {
System.out.println(response.statusCode());
async.complete();
});
//
// Async async = tc.async();
// vertx.createHttpClient().getNow(8080, "localhost", "/", response -> {
// tc.assertEquals(response.statusCode(), 200);
// response.bodyHandler(body -> {
// tc.assertTrue(body.length() > 0);
// async.complete();
// });
// });
}
}

Vert.x unit test doesn't execute

I'm new to vert.x and decided to start out light, but can't get a simple unit test to work (using maven to build). Code as follows
FileRepo.java:
import io.vertx.core.AbstractVerticle;
import io.vertx.core.Future;
import io.vertx.core.http.HttpServer;
import io.vertx.core.http.HttpServerResponse;
import io.vertx.ext.web.Router;
public class FileRepo extends AbstractVerticle {
#Override
public void start(Future<Void> fut) {
HttpServer server = vertx.createHttpServer();
Router router = Router.router(vertx);
router.route("/upload").handler(routingContext -> {
HttpServerResponse response = routingContext.response();
response.putHeader("content-type", "text/plain");
response.end("Hello world!");
});
System.out.println("Starting server!");
server.requestHandler(router::accept).listen(8080);
System.out.println("Server started!");
}
}
FileRepoTest.java:
import FileRepo;
import io.vertx.core.Vertx;
import io.vertx.ext.unit.TestContext;
import io.vertx.ext.unit.junit.VertxUnitRunner;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
#RunWith(VertxUnitRunner.class)
public class FileRepoTest {
private Vertx vertx;
#Before
public void setUp(TestContext context) {
vertx = Vertx.vertx();
System.out.println("SetUp Vertx");
vertx.deployVerticle(FileRepo.class.getName(), context.asyncAssertSuccess());
System.out.println("SetUp done");
}
#After
public void tearDown(TestContext context) {
System.out.println("tearDown Vertx");
vertx.close(context.asyncAssertSuccess());
}
#Test
public void testUpload(TestContext context) {
System.out.println("testUpload");
}
}
Result:
SetUp Vertx
SetUp done
Starting server!
Server started!
tearDown Vertx
java.util.concurrent.TimeoutException
at io.vertx.ext.unit.impl.TestContextImpl$Step.lambda$run$0(TestContextImpl.java:112)
at java.lang.Thread.run(Thread.java:745)
Process finished with exit code -1
Browsing http://localhost:8080/upload while waiting for the TimeoutException shows a Hello World! page, but the #Test never seem to execute. What am I doing wrong here?
Regards,
Mattias

The exception you are getting is because there is no acknowledgment of the server start state.
Change your Verticle to following:
import io.vertx.core.AbstractVerticle;
import io.vertx.core.Future;
import io.vertx.core.http.HttpServerResponse;
import io.vertx.ext.web.Router;
public class FileRepo extends AbstractVerticle {
#Override
public void start(Future<Void> fut) {
Router router = Router.router(vertx);
router.route("/upload").handler(routingContext -> {
HttpServerResponse response = routingContext.response();
response.putHeader("content-type", "text/plain");
response.end("Hello world!");
});
System.out.println("Starting server!");
vertx.createHttpServer()
.requestHandler(router::accept)
.listen(8080, result -> {
if (result.succeeded()) {
System.out.println("Server started!");
fut.complete();
} else {
System.out.println("Server start failed!");
fut.fail(result.cause());
}
});
}
}

How to run jetty server for java junit testing

I am trying to write an integration test to spin up jetty server locally and then use client to communicate with the rest URI and call the business logic downstream. However, when I start my jetty server it does not relinquish the control, so my client is not executed. So I used threading to start my jetty in a different thread, however, the thread finishes before my client call, it says connection refused. Any approach I can take?
#Test
public void testPerform() {
final JettyServer jettyServer = JettyServer.create();
jettyServer.buildJettyServer(ServletContextHandler.SESSIONS, "/", 8080, TestResource.class);
Runnable runnable = new Runnable()
{
#Override
public void run()
{
jettyServer.start();
}
};
new Thread(runnable).start();
final javax.ws.rs.client.Client client = ClientBuilder.newClient();
final Response response = client.target("http://localhost:8080/test").request().post(Entity.text(""));
jettyServer.stop();
}

Skip the Runnable, skip the new Thread(runnable).start()
The call jettyServer.start() starts the server on its own thread (along with all of the other threads that the server needs.
For a basic example of junit and jetty ...
#Test
public void testGet() throws Exception
{
// Create Server
Server server = new Server(8080);
ServletContextHandler context = new ServletContextHandler();
ServletHolder defaultServ = new ServletHolder("default", DefaultServlet.class);
defaultServ.setInitParameter("resourceBase",System.getProperty("user.dir"));
defaultServ.setInitParameter("dirAllowed","true");
context.addServlet(defaultServ,"/");
server.setHandler(context);
// Start Server
server.start();
// Test GET
HttpURLConnection http = (HttpURLConnection)new URL("http://localhost:8080/").openConnection();
http.connect();
assertThat("Response Code", http.getResponseCode(), is(HttpStatus.OK_200));
// Stop Server
server.stop();
}
The #Before and #After junit annotations can also be used. This will start the server before each #Test and stop the server after.
package jetty;
import static org.hamcrest.Matchers.*;
import static org.junit.Assert.*;
import java.net.HttpURLConnection;
import java.net.URL;
import org.eclipse.jetty.http.HttpStatus;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.servlet.DefaultServlet;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
public class JUnitBeforeAfterJettyTest
{
private Server server;
#Before
public void startJetty() throws Exception
{
// Create Server
server = new Server(8080);
ServletContextHandler context = new ServletContextHandler();
ServletHolder defaultServ = new ServletHolder("default", DefaultServlet.class);
defaultServ.setInitParameter("resourceBase",System.getProperty("user.dir"));
defaultServ.setInitParameter("dirAllowed","true");
context.addServlet(defaultServ,"/");
server.setHandler(context);
// Start Server
server.start();
}
#After
public void stopJetty()
{
try
{
server.stop();
}
catch (Exception e)
{
e.printStackTrace();
}
}
#Test
public void testGet() throws Exception
{
// Test GET
HttpURLConnection http = (HttpURLConnection)new URL("http://localhost:8080/").openConnection();
http.connect();
assertThat("Response Code", http.getResponseCode(), is(HttpStatus.OK_200));
}
}
For the best approach, you can also use the #BeforeClass and #AfterClass techniques, along with auto-binding to an open port. This will only start the server once, per Test Class, run all of the #Test methods, then stop the server once at the end.
package jetty;
import static org.hamcrest.Matchers.*;
import static org.junit.Assert.*;
import java.net.HttpURLConnection;
import java.net.URI;
import java.net.URL;
import org.eclipse.jetty.http.HttpStatus;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import org.eclipse.jetty.servlet.DefaultServlet;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
public class JUnitBeforeAfterClassJettyTest
{
private static Server server;
private static URI serverUri;
#BeforeClass
public static void startJetty() throws Exception
{
// Create Server
server = new Server();
ServerConnector connector = new ServerConnector(server);
connector.setPort(0); // auto-bind to available port
server.addConnector(connector);
ServletContextHandler context = new ServletContextHandler();
ServletHolder defaultServ = new ServletHolder("default", DefaultServlet.class);
defaultServ.setInitParameter("resourceBase",System.getProperty("user.dir"));
defaultServ.setInitParameter("dirAllowed","true");
context.addServlet(defaultServ,"/");
server.setHandler(context);
// Start Server
server.start();
// Determine Base URI for Server
String host = connector.getHost();
if (host == null)
{
host = "localhost";
}
int port = connector.getLocalPort();
serverUri = new URI(String.format("http://%s:%d/",host,port));
}
#AfterClass
public static void stopJetty()
{
try
{
server.stop();
}
catch (Exception e)
{
e.printStackTrace();
}
}
#Test
public void testGet() throws Exception
{
// Test GET
HttpURLConnection http = (HttpURLConnection) serverUri.resolve("/").toURL().openConnection();
http.connect();
assertThat("Response Code", http.getResponseCode(), is(HttpStatus.OK_200));
}
}