How to upload Json Data or file in Elasticsearch using Java? - java

This is my sample Json Data coming from .json file now I want to do bulk_insert to elasticsearch dynamically so that I can perform operations on it ..can someone help me with java code to add this data dynamically ..this is just a piece of 5-6objects like this i have more then 500objects
[{
"data1" : "developer",
"data2" : "categorypos",
"data3" : "1001"
},
{
"data1" : "developer",
"data1" : "developerpos",
"data1" : "1002"
},
{
"data1" : "developer",
"data2" : "developpos",
"data3" : "1003"
},
{
"data1" : "support",
"data2" : "datapos",
"data3" : "1004"
}
]

There is a provision of bulk operations in elastic search following is the documentation this might help
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html

You need to read the file from your application, iterate over the array and for each document, send it to elasticsearch.
To do the latest, you should use the bulk processor class.
BulkProcessor bulkProcessor = BulkProcessor.builder(
(request, bulkListener) -> esClient.bulkAsync(request, RequestOptions.DEFAULT, bulkListener),
new BulkProcessor.Listener() {
#Override
public void beforeBulk(long executionId, BulkRequest request) { }
#Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse response) { }
#Override
public void afterBulk(long executionId, BulkRequest request, Throwable failure) { }
})
.setBulkActions(10000)
.setFlushInterval(TimeValue.timeValueSeconds(5))
.build();
For each json document, call:
bulkProcessor.add(new IndexRequest("INDEXNAME").source(json, XContentType.JSON));

Related

Azure Functions CosmosDb Binding api keeps loading

First of all, the api works as intended locally, when deploying to azure functions app, the api endpoint keeps loading and it will eventually show HTTP.504(Gateway Timeout)
page keeps loading, no response from azure functions
Integration
I'm looking to fetch all data from the collection when I call HttpTrigger
Function.java
#FunctionName("get")
public HttpResponseMessage get(
#HttpTrigger(name = "req",
methods = {HttpMethod.GET, HttpMethod.POST},
authLevel = AuthorizationLevel.ANONYMOUS)
HttpRequestMessage<Optional<String>> request,
#CosmosDBInput(name = "database",
databaseName = "progMobile",
collectionName = "news",
partitionKey = "{Query.id}",
connectionStringSetting = "CosmosDBConnectionString")
Optional<String> item,
final ExecutionContext context) {
// Item list
context.getLogger().info("Parameters are: " + request.getQueryParameters());
context.getLogger().info("String from the database is " + (item.isPresent() ? item.get() : null));
// Convert and display
if (!item.isPresent()) {
return request.createResponseBuilder(HttpStatus.BAD_REQUEST)
.body("Document not found.")
.build();
}
else {
// return JSON from Cosmos. Alternatively, we can parse the JSON string
// and return an enriched JSON object.
return request.createResponseBuilder(HttpStatus.OK)
.header("Content-Type", "application/json")
.body(item.get())
.build();
}
}
Function.json
{
"scriptFile" : "../ProgMobileBackend-1.0-SNAPSHOT.jar",
"entryPoint" : "com.function.Function.get",
"bindings" : [ {
"type" : "httpTrigger",
"direction" : "in",
"name" : "req",
"methods" : [ "GET", "POST" ],
"authLevel" : "ANONYMOUS"
}, {
"type" : "cosmosDB",
"direction" : "in",
"name" : "database",
"databaseName" : "progMobile",
"partitionKey" : "{Query.id}",
"connectionStringSetting" : "CosmosDBConnectionString",
"collectionName" : "news"
}, {
"type" : "http",
"direction" : "out",
"name" : "$return"
} ]
}
Azure Functions monitor log does not show any error
Running the function in the portal(Code + Test menu) does not show any error either
httpTrigger I'm using: https://johnmiguel.azurewebsites.net/api/get?id=id
I added CosmosDBConnectionString value to Azure Functions App configuration(did not check on "Deployment slot" option)
I'm using an instance of CosmosDB for NoSQL
Functions App runtime is set to Java and version set to Java 8
figured it out. Java function was in Java 17 and Function App in Java 8.

Wiremock - Autoconfigure doesn't load stubs

My wiremock doesn't seem to work with autoconfigure. I have json files in a folder named stubs in the classpath and I ran the standalone jar on the port 8080.
#AutoConfigureWireMock(stubs="classpath:/stubs", port = 0)
public class TestResource {
#Autowired
private Service service;
#Test
public void contextLoads() throws Exception {
assertThat(this.service.go()).isEqualTo("Hello World!");
}
}
Example of a json file
{
"request" : {
"url" : "/api/users",
"method" : "GET",
"bodyPatterns" : [ {
"contains" : "some soap body"
}]
},
"response" : {
"status" : 200,
"body" : "Hello World",
"headers" : {
"X-Application-Context" : "application:-1",
"Content-Type" : "text/plain"
}
}
}
When I launch a request with GET -> localhost:8080/api/users/ It doesn't match with the json file.
Thanks in advance
I simply added all my json with a POST request on localhost:8080/__admin/mappings/import
http://wiremock.org/docs/stubbing/

Wiremock error - there are no stub mappings in this WireMock instance

I have implemented a basic WireMock with a sample REST/HTTP request simulation. The server code implemented as below.
With this code, I get the following error when I issue the GET request from Postman (i.e. GET http://127.0.0.1:8089/some/thing).
No response could be served as there are no stub mappings in this WireMock instance.
What is missing in my setup/code?
import com.github.tomakehurst.wiremock.WireMockServer;
import com.github.tomakehurst.wiremock.core.WireMockConfiguration;
public class MockApp {
private WireMockServer wireMockServer;
public MockApp(String testSpec) {
wireMockServer = new WireMockServer(WireMockConfiguration.options().
port(8089).
usingFilesUnderDirectory(testSpec).
disableRequestJournal());
}
public void start() {
wireMockServer.start();
}
public void stop() {
wireMockServer.stop();
}
}
The main function is:
public class MockMain {
public static void main(String[] args) {
String baseDir = System.getProperty("user.dir");
String testResource = baseDir + "/resources/testconfig/";
MockAMS mockAMS = new MockAMS(testResource);
mockAMS.start();
}
}
Under 'resources/testconfig', there is a file called mapping.json containing:
{
"request": {
"method": "GET",
"url": "/some/thing"
},
"response": {
"status": 200,
"body": "Hello world!",
"headers": {
"Content-Type": "text/plain"
}
}
}
I found solution for this. So, basically we need create a folder called "mappings" (exact name) under the directory identified by "testResource" variable. So in above code example, the mapping.json file will be stored at location: "MockApp/resources/testconfig/mappings/mapping.json".
Once this, is done, it will print the following output. As can be seen in the logs, "Stub mapping size is 1". This will be printed once you add the following line in the code.
System.out.println("Stub mapping size: " + wireMockServer.getStubMappings().size());
Stub mapping size: 1
{
"id" : "da5735a6-b6cc-45aa-8256-fb88b5670610",
"request" : {
"url" : "/some/thing",
"method" : "GET"
},
"response" : {
"status" : 200,
"body" : "Hello world!",
"headers" : {
"Content-Type" : "text/plain"
}
},
"uuid" : "da5735a6-b6cc-45aa-8256-fb88b5670610"
}

Elastic Search autocompletion/suggestion Java

I want to implement a search as you type functionality in Elastic Search using Java API. The queries that I want to transform to Java are below.
Do you have any idea, how can I execute this queries in Java?
These queries are very similar but I want to solve at least one.
This is my initial approach:
SearchResponse response = client.prepareSearch("kal")
.setTypes("products")
.setQuery(multiMatchQuery("description_en", "name", "description_en"))// Query
.setFrom(0).setSize(60).setExplain(true)
.get();
SearchHit[] results = response.getHits().getHits();
for (SearchHit hit : results) {
String sourceAsString = hit.getSourceAsString();
Map<String, SearchHitField> responseFields = hit.getFields();
SearchHitField field = responseFields.get("product_id");
Map map = hit.getSource();
System.out.println(map.toString());
}
Queries:
POST /kal/products/_search?pretty
{
"suggest": {
"name-suggest" : {
"prefix" : "wine",
"completion" : {
"field" : "suggest_name"
}
}
}
}
GET /kal/products/_search
{ "query": {
"prefix" : {
"name" : "wine",
"description": "wine"
}
}
}
GET /kal/products/_search
{
"query" : {
"multi_match" : {
"fields" : ["name", "description_en"],
"query" : "description_",
"type" : "phrase_prefix"
}
}
}

How to create Spring Data Rest entities response format manually [duplicate]

This question already has answers here:
Calling spring data rest repository method doesn't return links
(2 answers)
Closed 6 years ago.
I'm using Spring Data Rest to create a RESTful api. I want to handle an exception returning an entity representation like the ones generated by the Spring Data Rest repositories (with HATEOAS links). The method from where I need to return the entity representation is the following:
#ExceptionHandler(value = {ExistentUGVException.class})
#ResponseBody
protected ResponseEntity<UGV> existentUGVHandler(HttpServletRequest request, HttpServletResponse response, ExistentUGVException ex) {
return new ResponseEntity<UGV>(ex.ugv, HttpStatus.OK);
}
This implementation returns the UGV representation without links:
{
"title" : "Golden Eagle Snatches Kid",
"publishDate" : "2012-12-19T13:55:28Z",
"url" : "https://www.youtube.com/watch?v=Xb0P5t5NQWM"
}
But it would be:
{
"title" : "Golden Eagle Snatches Kid",
"publishDate" : "2012-12-19T13:55:28Z",
"url" : "https://www.youtube.com/watch?v=Xb0P5t5NQWM",
"_links" : {
"self" : {
"href" : "http://localhost/youTubeVideos/Xb0P5t5NQWM"
},
"youTubeVideo" : {
"href" : "http://localhost/youTubeVideos/Xb0P5t5NQWM{?projection}",
"templated" : true
},
"user" : {
"href" : "http://localhost/youTubeVideos/Xb0P5t5NQWM/user"
}
}
}
You'll have to transform your ResponseEntity to Resource first and then add the links manually.
It should be something like this :
#ExceptionHandler(value = {ExistentUGVException.class})
#ResponseBody
protected ResponseEntity<Resource<UGV>> existentUGVHandler(HttpServletRequest request, HttpServletResponse response, ExistentUGVException ex) {
final Resource<UGV> resource = getResource(ex.ugv);
return new ResponseEntity<Resource<UGV>>(resource, HttpStatus.OK);
}
public Resource<T> getResource(T object, Link... links) throws Exception {
Object getIdMethod = object.getClass().getMethod("getId").invoke(object);
Resource<T> resource = new Resource<T>(object); // The main resource
final Link selfLink = entityLinks.linkToSingleResource(object.getClass(), getIdMethod).withSelfRel();
String mappingRel = CLASSMAPPING.getMapping(this.getClass());
final Link resourceLink = linkTo(this.getClass()).withRel(mappingRel);
resource.add(selfLink, resourceLink);
resource.add(links);
return resource;
}
Take a look here, there's all you need : spring hateoas documentation

Categories