We Keep Coding

loading data to Elasticsearch v7.3 using Bulk API

I need to load the data to the elasticsearch index. I am using BULK API of elasticsearch to load the JSONs to index.
private String FOLDER_PATH = "src/main/resources/allJsons";
private String index = "test1";
private static final String TYPE = "test_type";
#Autowired
private RestHighLevelClient restHighLevelClient;
public String loadBulkData() throws IOException {
BulkRequest bulkRequest = new BulkRequest();
AtomicInteger counter = new AtomicInteger();
try (Stream<Path> filePathStream = Files.walk(Paths.get(FOLDER_PATH))) {
filePathStream.forEach(filePath -> {
if (Files.isRegularFile(filePath)) {
counter.getAndIncrement();
try {
String content = Files.readString(filePath);
JSONObject jsonObject1 = new JSONObject(content);
HashMap yourHashMap1 = new Gson().fromJson(jsonObject1.toString(), HashMap.class);
IndexRequest indexRequest = new IndexRequest(index, TYPE).source(yourHashMap1);
bulkRequest.add(indexRequest);
} catch (IOException e) {
e.printStackTrace();
}
}
});
}
try {
restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
e.printStackTrace();
}
return "Bulk data loaded to index " + index + "";
}
}
I have multiple JSONs based on the following format
[
{
"Nutrient" : "Calories",
"Amount" : " 289.00",
"Unit" : " kcal"
}, {
"Nutrient" : "Fat",
"Amount" : " 17.35",
"Unit" : " g"
}
]
While running the code it gives me error ,
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.json.JSONException: A JSONObject text must begin with '{' at 1 [character 2 line 1]
I think the data is in JSONArray and for the code, we need JSONObject. Anyone could please guide to how to do this

You can do bulk insertion by passing hashmap of your json objects to Elasticsearch Bulk API.
You can create Hashmap by parsing your JSON file through JSONParser.
Here is the code for the same :
Code :
Integer id= 1;
//You need to call this method for inserting bulk documents which
// internally calls `createBulkRequest` and `parseObjectList` methods.
//This method uses JSONParser to parse your file and convert into JSONArray.
public String insertBulkDocuments() throws Exception {
Object obj = new JSONParser().parse(new FileReader(<path-of-file>));
JSONArray objList= (JSONArray) obj;
BulkRequest request = createBulkRequest(objList);
BulkResponse bulkresp=restHighLevelClient.bulk(request, RequestOptions.DEFAULT);
return bulkresp.status().toString();
}
// Each JSONArray element that was obtained through first method
//is parsed individually through Gson and converted into you defined Object.
//This object is then converted to Map and passed to IndexRequest object.
private BulkRequest createBulkRequest(JSONArray objList) {
BulkRequest request = new BulkRequest();
objList.forEach( obj -> parseObjectList((JSONObject) obj, request,id++));
return request;
}
private void parseObjectList(JSONObject obj, BulkRequest request, int id) {
Gson gson = new GsonBuilder().create();
NutrientDocument doc = gson.fromJson(obj.toJSONString(), NutrientDocument .class);
Map<String, Object> documentMapper = objectMapper.convertValue(doc, Map.class);
IndexRequest indexRequest = new IndexRequest(<your-index-name>).id(Integer.toString(id)).source(documentMapper);
request.add(indexRequest);
}
You need to create Custom object which has same feilds as your json . I have created NutrientDocument for testing which has same fields as your JSON and this I am using in parseObjectList method.
public class NutrientDocument {
private String Nutrient;
private Float Amount;
private String Unit;
public String getNutrient() {
return Nutrient;
}
public void setNutrient(String nutrient) {
Nutrient = nutrient;
}
public Float getAmount() {
return Amount;
}
public void setAmount(Float amount) {
Amount = amount;
}
public String getUnit() {
return Unit;
}
public void setUnit(String unit) {
Unit = unit;
}
}
NOTE :
For each document elasticserach generates unique id .
For creating our own id value instead of Elasticsearch autogenerated value, we are using id variable. But, if you want to go with Elasticsearch autogenerated number , you can create IndexRequest object as below in parseObjectList method and remove id variable wherever we are passing.
IndexRequest indexRequest = new IndexRequest().source(documentMapper);

Convert JSON to JAVA object using Jackson keeps running, no error

I'm trying to convert a JSON (from REST API) to a Java Object using Jackson.
But when I run this code. Netbeans keeps running but nothing changes..
My hash works fine, so I don't think my URL is wrong..
Is my mapper wrongly configured? I can't find a solution..
public Beheerder meldAan(String email, String wachtwoord){
if (email == null || wachtwoord == null) {
throw new IllegalArgumentException("Gebruikersnaam en wachtwoord moeten worden ingevuld.");
}
Beheerder beheerder = null;
try {
List<Beheerder> beheerders = repo.geefAlleBeheerders();
if (controlleerBeheerder(beheerders, email)) {
throw new IllegalArgumentException("Gebruikersnaam is verkeerd ingevuld.");
}
String hash = sha256(wachtwoord);
//URL jsonUrl = new URL("https://studservice.hogent.be/auth/" + email + "/" + hash);
ObjectMapper mapper = new ObjectMapper();
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
beheerder = mapper.readValue(jsonUrl, Beheerder.class);
if (beheerder == null) {
throw new IllegalArgumentException("Gelieve een correct wachtwoord in te geven");
}
return beheerder;
} catch (MalformedURLException e) {
e.printStackTrace();
} catch(IOException e){
e.printStackTrace();
}
return beheerder;
}
public class Beheerder {
private int beheerderId;
private String faculteit;
private String naam;
private String foto;
private String type;
private String voornaam;
private String email;
private boolean isHoofdbeheerder;
public Beheerder(String f, String fm, String foto, String type, String vm, String email){
this.faculteit = f;
this.type = type;
this.naam = fm;
this.foto = foto;
this.voornaam = vm;
this.email = email;
} //With some getters and setters..

I assume there is a problem retrieving the data from the remote server.
You should separate these concerns: Have one method fetching the data and one method deserializing it. You can test the JSON mapping then separately, f.e. with a String input from a unit test or a separate main class.
Try loading the JSON manually from your browser or command line with curl/wget as well. (Could be a firewall issue which would explain that it takes so long instead of immediate failure.)

In the picture you can see the variables when I debugged it..
Debug session
String hash = sha256(wachtwoord);
String json = "{\"FACULTEIT\":\"FBO\",\"NAAM\":\"De Durpel\",\"BASE64FOTO\":\"zezz//\",\"TYPE\":\"student\",\"VOORNAAM\":\"Kas\",\"EMAIL\":\"kas.dedurpel.s3007#student.ho‌​gent.be\"}";
ObjectMapper mapper = new ObjectMapper();
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
beheerder = mapper.readValue(json, Beheerder.class);
if (beheerder == null) {
throw new IllegalArgumentException("Gelieve een correct wachtwoord in te geven");
}
My 'Beheerder' object stays null when I do it manually.
But It is a problem retrieving the data I guess!

where is the error in this RowMapper

I have this Request SQL :
private List<IWsResponse> getBPartnerDetails(String valueNetwork, String reportLevel2) {
JdbcTemplate tm = new JdbcTemplate(ds.getDataSource());
StringBuffer sql = new StringBuffer("SELECT * FROM XRV_BPARTNERDETAILS order by BPartner_ID");
ArrayList<Object> params = new ArrayList<Object>();
response = tm.query(sql.toString(), new BPartnerMapper());
return response;
}
i create a new RowMapper (BPartnerMapper) like this
public class BPartnerMapper implements RowMapper<IWsResponse> {
#Override
public List<IWsResponse> mapRow(ResultSet rs, int rowNum) throws SQLException {
List<IWsResponse> bps = new ArrayList<IWsResponse>();
while (rs.next()) {
bp = new BPartner();
bp.setBPartnerValue(rs.getString("BPartnerValue"));
//adress
adr = new Adress();
adr.setBPartnerLocation_ID(BPartner_Location_ID);
bp.getAdress().add(adr);
//user
usr = new User();
usr.setUser_ID(User_ID);
bp.getUsers().add(usr);
bps.add(bp)
}
return bps;
}
the Class BPartner is
public class BPartner implements IWsResponse {
private String BPartnerValue;
private ArrayList<Adress> adress = new ArrayList<Adress>();
private ArrayList<User> users = new ArrayList<User>();
}
so i get this Error
The return type is incompatible with RowMapper<IWsResponse>.mapRow(ResultSet, int)

Take a look at the documentation. A row mapper maps objects for a single row.
This method should not call next() on the ResultSet; it is only supposed to map values of the current row.
So your return type should be IWsResponse rather than List. Additionally you must only map the current row. Not all of them.

How to cast to your Model object at reading from Mongo instead reading DBObject?

I wondering to know how to read saced object from MongoDB.
I just saved it with key (dataStatus) and object was converted to JSON.
When I tryng to retrieve it - I have DBObject instead of my model.
easy main():
public static void main(String[] args) {
LowFareSearchRQDao searchRQDao = new LowFareSearchRQDao();
searchRQDao.connect();
System.out.println(ToStringBuilder.reflectionToString(searchRQDao.read(DataStatus.SUCCESS_LOW_FARE_SEARCH_REQUEST), ToStringStyle.MULTI_LINE_STYLE));
searchRQDao.disconnect();
}
Output:
com.mongodb.BasicDBObject#76e134e6[
_isPartialObject=false
accessOrder=false
threshold=12
loadFactor=0.75
]
Here is my create():
#Override
public void create(MODEL model) {
try {
Field keyField1 = getKeyField(model);
String fieldValue = getKeyFieldValue(keyField1, model);
BasicDBObject query = createQuery(keyField1.getName(), fieldValue);
DBCursor cursor = dbCollection.find(query);
if (!cursor.hasNext()) {
DBObject dbObject = getDbObject(model);
dbCollection.insert(dbObject);
} else {
throw new RuntimeException(String.format("Duplicate data status %s with value %s", keyField1.getName(), fieldValue));
}
} catch (IOException e) {
e.printStackTrace();
}
}
private Field getKeyField(MODEL model) {
Field keyField = null;
for (Field field : model.getClass().getDeclaredFields()) {
if (field.isAnnotationPresent(KeyField.class)) {
keyField = field;
}
}
if (keyField == null) {
throw new RuntimeException(String.format("Couldn't find KeyField annotation at class '%s'", model.getClass().getName()));
}
return keyField;
}
private String getKeyFieldValue(Field keyField, Object model) {
String result = null;
try {
if(keyField.isAnnotationPresent(KeyField.class)) {
keyField.setAccessible(true);
result = keyField.get(model).toString();
}
if(result == null || result.equals("")) {
throw new NullPointerException();
}
} catch(NullPointerException e) {
throw new RuntimeException("KeyField property is empty");
}catch (Exception e) {
throw new RuntimeException(String.format("Couldn't find KeyField annotation at class '%s'", model.getClass().getName()));
}
return result;
}
private BasicDBObject createQuery(String key, String value) {
BasicDBObject query = new BasicDBObject();
query.put(key, value);
return query;
}
#Override
#SuppressWarnings("unchecked")
public MODEL read(ID id) {
Field keyField2 = getKeyField(model);
BasicDBObject query = createQuery(keyField2.getName(), id.toString());
DBCursor cursor = dbCollection.find(query);
if (!cursor.hasNext()) {
throw new RuntimeException(String.format("This id %s isn't presented at collection %s", id.toString(), dbCollection.getFullName()));
}
return (MODEL) JSON.parse(String.valueOf(cursor.next()));
}
How to convert from DBObject to your model object?

How to convert from DBObject to your model object?
This is one way where you can achieve it in a simplified manner:
Make your class extend BasicDBObject, with a parameterized constructor.
class Model extends BasicDBObject
{
public Model(DBObject object) {
super(object.toMap());
}
}
Setting values in a Model.
Model model = new Model();
model.put("key","value");
Inserting a Model into the database.
dbCollection.insert(model);
Retrieving a Model from the database.
while(cursor.hasNext()){
DbObject obj = cursor.next();
Model model = new Model(obj);
//do something with the Model
}

I found solution.
For this purpose I used ObjectMapper. And I want to skip when my objet contains unknown properties as _id.
Here is code snippet:
#SuppressWarnings("unchecked")
public MODEL read(ID id) {
Field keyField2 = getKeyField(model);
BasicDBObject query = createQuery(keyField2.getName(), id.toString());
DBCursor cursor = dbCollection.find(query);
if (!cursor.hasNext()) {
throw new RuntimeException(String.format("This id %s isn't presented at collection %s", id.toString(), dbCollection.getFullName()));
}
String json = cursor.next().toString();
ObjectMapper mapper = new ObjectMapper();
mapper.configure(DeserializationConfig.Feature.FAIL_ON_UNKNOWN_PROPERTIES, false);
MODEL value = null;
try {
value = mapper.readValue(json, model);
} catch (IOException e) {
e.printStackTrace();
}
return value;
}

xStream changes order of objects in write/readObject with JSON serialization

We use XStream to serialize objects to JSON and vice versa.
We init xStream like this
XStream xStream = new XStream(new JettisonMappedXmlDriver(new Configuration(), false));
xStream.ignoreUnknownElements();
xStream.setMode(XStream.XPATH_RELATIVE_REFERENCES);
We have test class
public static class TestWOWithBI implements Serializable{
private static final long serialVersionUID = -4720678317857471031L;
private transient String customerNickname;
private transient String customerUuid;
private transient BigInteger discussionId;
private transient String message;
public TestWOWithBI(String customerNickname, String customerUuid, BigInteger discussionId, String message){
this.customerNickname = customerNickname;
this.customerUuid = customerUuid;
this.discussionId = discussionId;
this.message = message;
}
private final void writeObject(final ObjectOutputStream out) throws IOException {
out.defaultWriteObject();
out.writeObject(customerNickname);
out.writeObject(customerUuid);
out.writeObject(discussionId);
out.writeObject(message);
}
private final void readObject(final ObjectInputStream in) throws IOException, ClassNotFoundException{
in.defaultReadObject();
customerNickname = (String) in.readObject();
customerUuid = (String) in.readObject();
discussionId = (BigInteger) in.readObject();
message = (String) in.readObject();
}
}
After serialization it looks like this:
{
"somethere.ObjectToJSONSerializerTest$TestWOWithBI": {
"#serialization": "custom",
"somethere.ObjectToJSONSerializerTest$TestWOWithBI": {
"default": "",
"string": ["name",
"uuid",
"message"],
"big-int": 1
}
}
}
and deserialization fails with class cast. It was on 1.3.1 and 1.4.7 versions. Looks like bug to me, but may be where is some settings?
UPD:
Seems like org.codehaus.jettison.mapped.MappedXMLStreamWriter.JSONPropertyObject#withProperty
if(old != null) {
JSONArray values;
// Convert an existing property to an array
// and append to the array
if (old instanceof JSONArray) {
values = (JSONArray)old;
} else {
values = new JSONArray();
values.put(old);
}
values.put(value);
object.put(property.getKey(), values);
} else if(getSerializedAsArrays().contains(property.getKey())) {
JSONArray values = new JSONArray();
values.put(value);
object.put(property.getKey(), values);
} else {
// Add the property directly.
object.put(property.getKey(), value);
}
It just group elements of same type.