I'm new to Elastic search. Started building a Spring boot application with Elastic search.
Using the latest ES version "elasticsearch-7.7.1" and for integration, I'm using below maven dependency:
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>7.7.1</version>
</dependency>
Added below configuration to my spring boot app:
#Configuration
public class ESConfig {
#Bean(destroyMethod = "close")
public RestHighLevelClient client() {
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(
RestClient.builder(new HttpHost("localhost")));
return restHighLevelClient;
}
}
Added below properties to application.yaml
elasticsearch:
host: localhost
Getting below Exception on Application startup:
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.elasticsearch.client.RestHighLevelClient]: Factory method 'client' threw exception; nested exception is java.lang.NoSuchFieldError: IGNORE_DEPRECATIONS
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622)
... 19 common frames omitted
Caused by: java.lang.NoSuchFieldError: IGNORE_DEPRECATIONS
at org.elasticsearch.client.RestHighLevelClient.<clinit>(RestHighLevelClient.java:1902)
at com.sbs.communicationcontrol.search.config.ESConfig.client(ESConfig.java:14)
Can anyone please help why this exception occurred?
After some R&D, fixed the issue by adding below two dependencies:
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
<version>7.7.1</version>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>7.7.1</version>
</dependency>
I encountered the same problem when I was upgrading to org.elasticsearch.client:elasticsearch-rest-high-level-client:7.8.1 from 6.8.5. The reason behind the problem is updating elasticsearch-rest-high-level-client alone to the latest version conflicts with some of its dependent elastic search dependencies which were brought into classpath as transitive dependencies by spring-boot. When I check with the dependency tree, I found that the org.springframework.boot:spring-boot:2.3.1.RELEASE dependency brings org.elasticsearch.client:elasticsearch-rest-client:7.6.2, org.elasticsearch:elasticsearch:7.6.2 and the 7.6.2 conflicts with 7.8.1.
An Excerpt code snippet from the RestHighLevelClient citing IGNORE_DEPRECATIONS Java Docs.
public class RestHighLevelClient implements Closeable {
....
....
/**
* Ignores deprecation warnings. This is appropriate because it is only
* used to parse responses from Elasticsearch. Any deprecation warnings
* emitted there just mean that you are talking to an old version of
* Elasticsearch. There isn't anything you can do about the deprecation.
*/
private static final DeprecationHandler DEPRECATION_HANDLER = DeprecationHandler.IGNORE_DEPRECATIONS;
.....
.....
}
The error itself indicates us to update all related elasticsearch libraries, though I couldn't find any out-of-box elastic search BOM to resolve this version conflicts, I have done the below workaround.
dependencies{
implementation 'org.elasticsearch.client:elasticsearch-rest-client:7.8.1'
implementation 'org.elasticsearch.client:elasticsearch-rest-client:7.8.1'
implementation 'org.elasticsearch:elasticsearch:7.8.1'
}
//Since version 7.6.2 is selected by rule, substituting the version 7.8.1 as below
configurations.all {
resolutionStrategy {
dependencySubstitution {
substitute module('org.elasticsearch.client:elasticsearch-rest-high-level-client') with module('org.elasticsearch.client:elasticsearch-rest-high-level-client:7.8.1')
substitute module('org.elasticsearch.client:elasticsearch-rest-client') with module('org.elasticsearch.client:elasticsearch-rest-client:7.8.1')
substitute module('org.elasticsearch:elasticsearch') with module('org.elasticsearch:elasticsearch:7.8.1')
}
}
}
You are not correctly initializing your elasticsearch client, can you try with below code:
Please note that I am using the version which accepts the host, port and http scheme and it works fine for me and is the standard for creating the client.
#Configuration
#Primary
public class ElasticsearchConfig {
/**
* Creates a Elasticsearch client from config
*
* #return Elasticsearch client
*/
#Bean(destroyMethod = "close")
public RestHighLevelClient client() {
RestHighLevelClient client = new RestHighLevelClient(
RestClient.builder(new HttpHost("localhost", 9500, "http")));
return client;
}
}
And use below config
elasticsearch.host=localhost
elasticsearch.port=9500
You can set the version for all of the Spring-boot:
ext {
set('elasticsearch.version', '6.2.0')
}
to avoid overriding it in multiple places.
Related
I have a small standard Spring Boot microservice which is Spring Data REST for a single Mongo DB collection. The document contains the UUID field. The collection is a capped collection due to requirements.
Now I can build and run the app locally on my machine, no problem with that. However, we run everything in Docker on production. The container with the service is built without any issues, but once it starts, there is the following error:
{"timestamp":"2023-01-13T15:10:01.592Z","message":"Servlet.service()
for servlet [dispatcherServlet] in context with path [] threw exception
[Request processing failed; nested exception is
org.springframework.core.convert.ConverterNotFoundException:
No converter found capable of converting from type [org.bson.types.Binary]
to type [java.util.UUID]] with root cause",
"component":"org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet]",
"level":"ERROR","stack_trace":
"org.springframework.core.convert.ConverterNotFoundException:
No converter found capable of converting from type [org.bson.types.Binary]
to type [java.util.UUID]
at org.springframework.core.convert.support.GenericConversionService
.handleConverterNotFound(GenericConversionService.java:322)
...
at org.springframework.data.mongodb.core.convert.MappingMongoConverter
.doConvert(MappingMongoConverter.java:1826)
...
at org.springframework.data.mongodb.repository.support.SimpleMongoRepository
.findAll(SimpleMongoRepository.java:444)
I have several questions actually:
Basically, how to handle it? Is it possible to handle using Spring configs?
Why SimpleMongoRepository.findAll() is called during the application start?
We have several applications with Mongo DB - they are good. What's wrong with Spring Data REST in this respect?
Why is the build OK, but when the container starts it fails?
Code samples are on Groovy just like the service (everything is public by default).
The repository is described as the following
#RepositoryRestResource( collectionResourceRel = 'events', path = 'events' )
interface EventRepository extends MongoRepository<EventMessage, String> { }
The collection configuration is the following
class CustomApplicationRunner implements ApplicationRunner {
#Autowired
MongoTemplate mongoTemplate
#Autowired
ApplicationProperties configuration
#Override
void run(ApplicationArguments args) throws Exception {
mongoTemplate.writeResultChecking = WriteResultChecking.EXCEPTION
if ( !mongoTemplate.collectionExists( EventMessage ) ) {
mongoTemplate.createCollection(EventMessage, new CollectionOptions(
configuration.mongoCollectionMaxSizeBytes,
configuration.mongoCollectionMaxDocuments,
configuration.mongoCappedCollection )
)
}
}
}
In application.yml the mongo configs are (on the development side)
mongodb:
uri: mongodb://mongodb:27017/development
auto-index-creation: true
uuid-representation: STANDARD
authentication-database: admin
The Spring Boot dependencies are
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'org.springframework.boot:spring-boot-starter-amqp'
implementation 'org.springframework.boot:spring-boot-starter-data-mongodb'
implementation 'org.springframework.boot:spring-boot-starter-integration'
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation 'org.springframework.boot:spring-boot-starter-data-rest'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
}
Versions: SpringBoot 2.7.7, Groovy 4.0.6
Now I did a little research.
Here is the JIRA issue for Mongo DB driver to support to/from UUID conversion for both Binary and BsonBinary. However, only BsonBinary is updated, apparently it is not used.
Here is Spring Data MongoDB issue for the same problem (or very close to mine). The author provided the solution, but I wonder if it is the only way. I'd like to reach the author for more details, but they have been inactive since then.
I can implement the solution from the above, but I'd like to figure out what's exactly is going on.
Thank you all in advance!
Working further on the problem I accidentally found a solution.
There was Spring Boot Starter Spring Integration dependency. However, nothing really from Spring Integration was used in the service.
There was also Jackson2JsonObjectMapper bean defined which wasn't used anywhere as well. Likely it was a leftover from the early stages of the service. Here I should note that the data class in question was annotated by #JsonDeserializer with the specific deserializer class which worked correct.
So removing of the bean and the Spring Integration dependency resulted in resolving the issue.
I have a Spring Boot REST service and some unit tests written for the data layer. I use the embedded MongoDB dependency to perform the basic CRUD tests for my Repository class:
public interface UserRepository extends MongoRepository<UserEntity, String> {
Optional<UserEntity> findByUsername(String username);
}
I load the data from a JSON file (located under test/java/resources/data) and with the help of an ObjectMapper instance, I load the data into the embedded DB before each test and drop the collection after it's completed:
#DataMongoTest
class UserRepositoryTest {
// the path to the JSON file
private final File USER_DATA_JSON = Paths.get("src", "test", "resources", "data", "UserData.json").toFile();
// used to load a JSON file into a list of Users
private final ObjectMapper objectMapper = new ObjectMapper();
#Autowired
private MongoTemplate mongoTemplate; // makes the interaction with the embedded MongoDB much easier
#Autowired
private UserRepository userRepository;
#BeforeEach
void setUp() throws IOException {
// deserialize the JSON file to an array of users
UserEntity[] users = objectMapper.readValue(USER_DATA_JSON, UserEntity[].class);
// load each user into embedded MongoDB
Arrays.stream(users).forEach(mongoTemplate::save);
}
#AfterEach
void tearDown() {
// drop the users collection
mongoTemplate.dropCollection("users");
}
#Test
void testFindAllSuccess() {
// WHEN
List<UserEntity> users = userRepository.findAll();
// THEN
assertEquals(2, users.size(), "findAll() should return 2 users!");
}
// other test methods
}
In my local environment, everything works just fine, no configuration is needed inside the application.properties file. But when I execute a build on Jenkins, the following error appears for all the Repository tests:
UserRepositoryTest > testFindAllSuccess() FAILED
java.lang.IllegalStateException at DefaultCacheAwareContextLoaderDelegate.java:132
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException at ConstructorResolver.java:800
Caused by: org.springframework.beans.factory.BeanCreationException at ConstructorResolver.java:658
Caused by: org.springframework.beans.BeanInstantiationException at SimpleInstantiationStrategy.java:185
Caused by: java.net.UnknownHostException at InetAddress.java:1642
Caused by: java.net.UnknownHostException at Inet6AddressImpl.java:-2
The build.gradle file dependencies are declared as follows:
implementation 'org.openapitools:openapi-generator-gradle-plugin:5.0.0'
implementation 'org.springframework.boot:spring-boot-starter'
implementation 'org.springframework.boot:spring-boot-starter-data-mongodb'
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation "io.springfox:springfox-boot-starter:3.0.0"
implementation('org.modelmapper:modelmapper:2.3.0')
testImplementation 'org.springframework.boot:spring-boot-starter-test'
testImplementation 'de.flapdoodle.embed:de.flapdoodle.embed.mongo'
testImplementation 'io.projectreactor:reactor-test'
compile 'io.springfox:springfox-swagger-ui:3.0.0'
annotationProcessor group: 'org.springframework.boot', name: 'spring-boot-configuration-processor'
compile 'org.mongodb:mongodb-driver-sync'
implementation 'org.springframework.boot:spring-boot-starter-security'
implementation 'org.springframework.security:spring-security-test'
// https://mvnrepository.com/artifact/io.jsonwebtoken/jjwt
implementation group: 'io.jsonwebtoken', name: 'jjwt', version: '0.9.1'
I assume Spring Boot is not identifying the embedded MongoDB anymore. How can I configure it as simple as possible?
I saw there is a possibility of using containerized MongoDB and test it using #TestContainers, but for now I need to fix this issue since every build fails due to the tests.
LATER EDIT: By activating the --debug option in Jenkins when running the build, I discovered the following cause:
java.net.UnknownHostException: dev096.dev.cloud.******.eu: dev096.dev.cloud.******.eu: Name or service not known
Do you know if I have to add that address to the known hosts to my local machine (etc/hosts) or it should be configured locally with profiles (localhost for development and dev096.dev.cloud.******.eu for production)?
I managed to reproduce the same error locally and, if you haven't solved it yet, this is my take.
Remove all MongoDB custom configurations/properties, then add the following class to your project:
#Configuration
public class MongoTestConfig {
#Bean
MongoProperties properties() {
MongoProperties mongoProperties = new MongoProperties();
mongoProperties.setPort(33333);
mongoProperties.setHost("localhost");
return mongoProperties;
}
}
Make sure you tag your test class (UserRepositoryTest) with these three annotations:
#RunWith(SpringRunner.class)
#DataMongoTest
#Import(MongoTestConfig.class)
#DataMongoTest ignores all other bean definitions so we force the test to include the configuration that we just created using the explicit #Import.
Now the test will be forced to run on localhost. Hope it's going to work for you as it did for me! 😁
If dev096.dev.cloud.******.eu is a host used only in production, then your production server needs to know that host; not your local PC, nor Jenkins.
Ideally, you'd run your Jenkins tests using a 'jenkins' Spring profile, and then define the localhost in application-jenkins.properties.
Unfortunately, none of your solutions worked for me, but I found out later that dev096.dev.cloud.******.eu is the Jenkins instance itself.
The fix was to add an entry in /etc/hosts/ such that the server's IP to be also recognized as dev096.dev.cloud.******.eu
I plan to use Cassandra to save data reactively. To do that, I wrote the following interface:
#Repository
public interface ContributorStatRepository extends ReactiveCrudRepository<ContributorStat, Long> {
Flux<ContributorStat> findByAkonId(String akonId);
}
The exception above is thrown:
com.example.sample.controller.ContributorStatControllerTest > shouldReturnBadRequestWithBlankContributorStat FAILED
java.lang.IllegalStateException
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException
Do you know why the appropiate bean for ContributorStatRepository is not being created?
I am using Spring boot 2.0.0.M7 and these dependencies:
dependencies {
compile('org.springframework.boot:spring-boot-starter-actuator')
compile('org.springframework.boot:spring-boot-starter-data-jpa')
compile('javax.xml.bind:jaxb-api:2.3.0')
compile('org.springframework.boot:spring-boot-starter-webflux')
compile('org.springframework.boot:spring-boot-starter-data-cassandra-reactive')
testCompile('org.springframework.boot:spring-boot-starter-test')
testCompile('io.projectreactor:reactor-test')
}
Updated:
Running test:
#Test
public void shouldReturnBadRequestWithBlankContributorStat() throws Exception {
requestPayload = mapper.writeValueAsString(new ContributorStatDTO());
this.mockMvc.perform(post(CONTRIBUTOR_STATS_ROUTE)
.contentType(MediaType.APPLICATION_JSON)
.accept(MediaType.APPLICATION_JSON)
.content(requestPayload)).andDo(print())
.andExpect(status().isBadRequest());
}
It seems you're using an #WebMvcTest annotated class to test this case (I'm not sure though, that part is missing from your question).
To test Spring WebFlux applications, you should use #WebFluxTest (see reference documentation). Even if you do, the ContributorStatRepository bean won't be there since the web test slice will only consider the web parts of your application and you usually need to mock this one with #MockBean.
I am upgrading Spring Boot from 1.3 to 1.5. For upgrading to 1.5 I have replaced
#SpringApplicationConfiguration(classes = TestConfig.class)
#WebIntegrationTest
with
#SpringBootTest(classes = TestConfig.class)
Also, I am using
#Value("${local.server.port}")
protected int port;
to get port number defined in application.properties file. I further use this port number to build a REST URL.
But after the upgrade I am getting the error below whereas the same works fine with 1.3 Spring Boot Test.
Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'local.server.port' in value "${local.server.port}"
at org.springframework.util.PropertyPlaceholderHelper.parseStringValue(PropertyPlaceholderHelper.java:174)
Am I missing any changes that I need to do for this to work.
You have to provide a value for webEnvironment. In your case DEFINED_PORT like this
#SpringBootTest(classes = App.class, webEnvironment = WebEnvironment.DEFINED_PORT)
public class YourTest {
#LocalServerPort // shorthand for #Value("${local.server.port}")
private Integer port;
...
}
For details see: https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-testing.html#boot-features-testing-spring-boot-applications
Adding another alternate solution which I had elsewhere.
I had configured
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
and
#RunWith(SpringRunner.class)
#SpringBootTest(classes = App.class, webEnvironment = WebEnvironment.RANDOM_PORT)
public class YourTest {
#LocalServerPort // shorthand for #Value("${local.server.port}")
private Integer port;
...
}
Thinking that was it, and still getting this error even when specifying web environment etc. My ${local.server.port} seemed to be always null.
After some time, I noticed that my Spring Boot startup message contained no notion of the port it was using, so apparently it really didn't listen to any port at all - which explained why it was null in the first place. Adding actual container implementation dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jetty</artifactId>
</dependency>
Caused this to appear on my logs:
2019-02-26 18:45:47.231 INFO 12504 --- [ main] o.s.b.web.embedded.jetty.JettyWebServer : Jetty started on port(s) 43132 (http/1.1) with context path '/'
after which local.server.port and #LocalServerPort would also work.
For me the problem was that there was alternative #Configuration class(es) in my other test(s) like this:
#Configuration
public class ReadPropertiesConfiguration {
#Bean
PropertyPlaceholderConfigurer propConfig() {
PropertyPlaceholderConfigurer placeholderConfigurer = new PropertyPlaceholderConfigurer();
placeholderConfigurer.setLocation(new ClassPathResource("application.properties"));
return placeholderConfigurer;
}
}
and #SpringBootApplication of the app was picking that up due to its #ComponentScan, and for some reason it resulted in this problem. When adding exclusion for those and/or replacing them with other solutions things started again to work without problems.
I don't know the root cause why this happens, but that might be your issue as well.
First make sure the property is correctly spelled in the properties file. As i did just few days back using spring-boot-1.5.2 & it works.
Or
You need to add
#PropertySource("classpath:application.properties")
to your class, so it will pick your configurations.
If you need different configurations for test you can add
#TestPropertySource(locations="classpath:test.properties")
Refer #Value not work on Spring Boot Test
I have an elasticsearch instance running locally.
I have a spring boot application.
In my application I have a service ServiceX which contains an elasticsearch repository which extends ElasticsearchRepository.
So
Service X contains
YRepository extends ElasticsearchRepository
I have an elasticsearch instance running locally.
My elastic search settings are
ELASTICSEARCH (ElasticsearchProperties)
spring.data.elasticsearch.properties.http.enabled=true
spring.data.elasticsearch.properties.host = localhost
spring.data.elasticsearch.properties.port = 9300
When the application is started an elasticsearch template is created.
The client that is used is a NodeClient.
The settings for the NodeClient are
"http.enabled" -> "true"
"port" -> "9300"
"host" -> "localhost"
"cluster.name" -> "elasticsearch"
"node.local" -> "true"
"name" -> "Human Robot"
"path.logs" -> "C:/dev/git/xxx/logs"
The name of the elasticsearch (Human Robot in this case), does not match the local elasticsearch instance running (Nikki in this case).
It looks like it
1. creates a new instance of logstash
2. creates an embedded instance of logstash.
I have searched through a lot of information but cannot find any documentation to help.
Could people please advise about what settings to use?
Thanks.
I believe that you do not want to use the NodeClient but the TransportClient unless you want your application to become part of the cluster
I believe you have the following dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artificatId>spring-boot-starter-data-elasticsearch</artificatId>
</dependency>
then you need to create some configuration class as follows:
#Configuration
#PropertySource(value = "classpath:config/elasticsearch.properties")
public class ElasticsearchConfiguration {
#Resource
private Environment environment;
#Bean
public Client client() {
TransportClient client = new TransportClient();
TransportAddress address = new InetSocketTransportAddress(
environment.getProperty("elasticsearch.host"),
Integer.parseInt(environment.getProperty("elasticsearch.port"))
);
client.addTransportAddress(address);
return client;
}
#Bean
public ElasticsearchOperations elasticsearchTemplate() {
return new ElasticsearchTemplate(client());
}
}
Also check ElasticSearch section of the Spring Boot guide, and especially the section about spring.data.elasticsearch.cluster-nodes if you put multiple comma seperated list of host port it will be generated a TransportClient instead, your choice
Try it, hope it helps
Thanks. Would you believe I actually just started trying to use a configuration file before I saw your post. I added a configuration class
#Configuration
public class ElasticSearchConfig {
#Bean
public Client client() {
TransportClient client = new TransportClient();
TransportAddress address = new InetSocketTransportAddress(
"localhost",9300);
client.addTransportAddress(address);
return client;
}
}
And the client is now being injected into the elasticsearch template (so don't need the elasticsearchtemplate bean).
I had an error when I tried to connect but that turned out to be due to elasticsearch 2.2.0, have tried it with elasticsearch 1.7.3 and it worked so now onto the next problem!