We Keep Coding

Indexing heavily nested json data in solr using DIH

I using DIH to import the data from a NoSQL data source . The format looks something like below
{
"id" : "123-145-app"
"name" : "apple",
"type" : "electronic",
"information": {
"category":["tablets","laptops","mobile"],
"stores": [
{
"name": "imagine",
"location" : "DLH"
},
{
"name": "abc",
"location" : "BLR"
}
],
"head_office" : "US"
}
}
when I try to index this using :
https://lucene.apache.org/solr/guide/6_6/transforming-and-indexing-custom-json.html
I am getting error stating : "Unknown operation for the an atomic update, operation ignored" , and the record with that data is getting skipped.
And in the documentation they have not mentioned about how to incorporate the split via configuration file or schema.xml
Can some one please help me with this ?

How to create a json file using existing json file as template?

I have a json file which is mostly standard for all my work, the only difference is few parameters.
Now I want to know how can I using java use this json file as a template and provide the parameters as input and save the new json file on local directory?
{
"key" : "HWM_NAME",
"value" : "PINE_SLS_SVC_CUST_CNTCT"
}, {
"key" : "TOPIC",
"value" : "SLS_SVC_CUST_CNTCT2"
}, {
"key" : "SRC_SCHEMA",
"value" : "party_pkg"
}, {
"key" : "SRC_TABLE",
"value" : "SLS_SVC_CUST_CNTCT"
}, {
"key" : "TGT_SCHEMA",
"value" : "mstrdata_hub"
}, {
"key" : "TGT_TABLE",
"value" : "SLS_SVC_CUST_CNTCT"
} ]
},
So here I wish to just change the Value: "PINE_SLS_SVC_CUST_CNTCT" to some other value that I would take as input from user and give me a new json file with those values.
PS: I am working on Java Swing to create a GUI to get the parameters from the user and provide the json file as output.enter image description here
this is how GUI looks

Consider using some JSON Library, for example GSon
Read the values from json
List<LinkedTreeMap> values = gson.fromJson(json, ArrayList.class);
...
update the values, here you can write values from UI components!!!!
for (LinkedTreeMap pair : values) {
//Update values
System.out.println(pair.toString());
}
And generate JSON from java objects
import com.google.gson.Gson;
import com.google.gson.internal.LinkedTreeMap;
...
Gson gson = new Gson();
String json = gson.toJson(values);

Update nested field in an index of ElasticSearch with Java API

I am using Java API for CRUD operation on elasticsearch.
I have an typewith a nested field and I want to update this field.
Here is my mapping for the type:
"enduser": {
"properties": {
"location": {
"type": "nested",
"properties":{
"point":{"type":"geo_point"}
}
}
}
}
Of course my enduser type will have other parameters.
Now I want to add this document in my nested field:
"location":{
"name": "London",
"point": "44.5, 5.2"
}
I was searching in documentation on how to update nested document but I couldn't find anything. For example I have in a string the previous JSON obect (let's call this string json). I tried the following code but seems to not working:
params.put("location", json);
client.prepareUpdate(index, ElasticSearchConstants.TYPE_END_USER,id).setScript("ctx._source.location = location").setScriptParams(params).execute().actionGet();
I have got a parsing error from elasticsearch. Anyone knows what I am doing wrong ?

You don't need the script, just update it.
UpdateRequestBuilder br = client.prepareUpdate("index", "enduser", "1");
br.setDoc("{\"location\":{ \"name\": \"london\", \"point\": \"44.5,5.2\" }}".getBytes());
br.execute();

I tried to recreate your situation and i solved it by using an other way the .setScript method.
Your updating request now would looks like :
client.prepareUpdate(index, ElasticSearchConstants.TYPE_END_USER,id).setScript("ctx._source.location =" + json).execute().actionGet()
Hope it will help you.

I am not sure which ES version you were using, but the below solution worked perfectly for me on 2.2.0. I had to store information about named entities for news articles. I guess if you wish to have multiple locations in your case, it would also suit you.
This is the nested object I wanted to update:
"entities" : [
{
"disambiguated" : {
"entitySubTypes" : [],
"disambiguatedName" : "NameX"
},
"frequency" : 1,
"entityType" : "Organization",
"quotations" : ["...", "..."],
"name" : "entityX"
},
{
"disambiguated" : {
"entitySubType" : ["a", "b" ],
"disambiguatedName" : "NameQ"
},
"frequency" : 5,
"entityType" : "secondTypeTest",
"quotations" : [ "...", "..."],
"name" : "entityY"
}
],
and this is the code:
UpdateRequest updateRequest = new UpdateRequest();
updateRequest.index(indexName);
updateRequest.type(mappingName);
updateRequest.id(url); // docID is a url
XContentBuilder jb = XContentFactory.jsonBuilder();
jb.startObject(); // article
jb.startArray("entities"); // multiple entities
for ( /*each namedEntity*/) {
jb.startObject() // entity
.field("name", name)
.field("frequency",n)
.field("entityType", entityType)
.startObject("disambiguated") // disambiguation
.field("disambiguatedName", disambiguatedNameStr)
.field("entitySubTypes", entitySubTypeArray) // multi value field
.endObject() // disambiguation
.field("quotations", quotationsArray) // multi value field
.endObject(); // entity
}
jb.endArray(); // array of nested objects
b.endObject(); // article
updateRequest.doc(jb);
Blblblblblblbl's answer couldn't work for me atm, because scripts are not enabled in our server. I didn't try Bask's answer yet - Alcanzar's gave me a hard time, because I supposedly couldn't formulate the json string correctly that setDoc receives. I was constantly getting errors that either I am using objects instead of fields or vice versa. I also tried wrapping the json string with doc{} as indicated here, but I didn't manage to make it work. As you mentioned it is difficult to understand how to formulate a curl statement at ES's java API.

A simple way to update the arraylist and object value using Java API.
UpdateResponse update = client.prepareUpdate("indexname","type",""+id)
.addScriptParam("param1", arrayvalue)
.addScriptParam("param2", objectvalue)
.setScript("ctx._source.field1=param1;ctx._source.field2=param2").execute()
.actionGet();
arrayvalue-[
{
"text": "stackoverflow",
"datetime": "2010-07-27T05:41:52.763Z",
"obj1": {
"id": 1,
"email": "sa#gmail.com",
"name": "bass"
},
"id": 1,
}
object value -
"obj1": {
"id": 1,
"email": "sa#gmail.com",
"name": "bass"
}

What is the best way to get longitude and latitude for an address via Google in java

I'm trying to write some java codes getting locations from address. (not in android environment.)
If possible, I want it to be permanent way as long as Google provides geo-location services in somehow.
I think I cannot say Google javascript geolocation apis uses permanent URL to do that, since the api's Object is to come from Google server at runtime.
However I think it seems to be possible that Android might use a permanent URL to do that since Google is not able to change every device's URLs through which devices get the geolocation services.
Am I wrong?
Is there any addressed Google policy concerning it?
Thanks in advance.

You may try to use the Google HTTP APIs for getting geocoding and reverse-geocoding.
Geocoding Requests
A Geocoding API request must be of the following form:
http://maps.googleapis.com/maps/api/geocode/output?parameters
where output may be either of the following values:
json (recommended) indicates output in JavaScript Object Notation (JSON)
xml indicates output as XML
Follow this link for more:https://developers.google.com/maps/documentation/geocoding/

If you are thinking to use Google GeoCoding then this stuff will be useful to you :
A Geocoding API request must be of the following form:
http://maps.googleapis.com/maps/api/geocode/output?parameters
where output may be either of the following values:
json (recommended) indicates output in JavaScript Object Notation (JSON)
xml indicates output as XML
To access the Geocoding API over HTTPS, use:
https://maps.googleapis.com/maps/api/geocode/output?parameters
HTTPS is recommended for applications that include sensitive user data, such as a user's location, in requests.
In either case, certain parameters are required while some are optional. As is standard in URLs, all parameters are separated using the ampersand (&) character. The list of parameters and their possible values are enumerated below.
Required parameters
address — The address that you want to geocode.
or
latlng — The textual latitude/longitude value for which you wish to obtain the closest, human-readable address. See Reverse Geocoding for more information.
or
components — A component filter for which you wish to obtain a geocode. See Component Filtering for more information. The components filter will also be accepted as an optional parameter if an address is provided.
sensor — Indicates whether or not the geocoding request comes from a device with a location sensor. This value must be either true or false.
Maps API for Business users must include valid client and signature parameters with their Geocoding requests. Please refer to Maps API for Business Web Services for more information.
Optional parameters
bounds — The bounding box of the viewport within which to bias geocode results more prominently. This parameter will only influence, not fully restrict, results from the geocoder. (For more information see Viewport Biasing below.)
language — The language in which to return results. See the list of supported domain languages. Note that we often update supported languages so this list may not be exhaustive. If language is not supplied, the geocoder will attempt to use the native language of the domain from which the request is sent wherever possible.
region — The region code, specified as a ccTLD ("top-level domain") two-character value. This parameter will only influence, not fully restrict, results from the geocoder. (For more information see Region Biasing below.)
components — The component filters, separated by a pipe (|). Each component filter consists of a component:value pair and will fully restrict the results from the geocoder. For more information see Component Filtering, below.
JSON Output Formats
In this example, the Geocoding API requests a json response for a query on "1600 Amphitheatre Parkway, Mountain View, CA":
http://maps.googleapis.com/maps/api/geocode/json?address=1600+Amphitheatre+Parkway,+Mountain+View,+CA&sensor=true_or_false
We've left the sensor parameter in this example as a variable true_or_false to emphasize that you must set this value to either true or false explicitly.
The JSON returned by this request is shown below. Note that actual JSON may contain less whitespace. You should not make assumptions about the amount or format of whitespace between requests.
{
"results" : [
{
"address_components" : [
{
"long_name" : "1600",
"short_name" : "1600",
"types" : [ "street_number" ]
},
{
"long_name" : "Amphitheatre Pkwy",
"short_name" : "Amphitheatre Pkwy",
"types" : [ "route" ]
},
{
"long_name" : "Mountain View",
"short_name" : "Mountain View",
"types" : [ "locality", "political" ]
},
{
"long_name" : "Santa Clara",
"short_name" : "Santa Clara",
"types" : [ "administrative_area_level_2", "political" ]
},
{
"long_name" : "California",
"short_name" : "CA",
"types" : [ "administrative_area_level_1", "political" ]
},
{
"long_name" : "United States",
"short_name" : "US",
"types" : [ "country", "political" ]
},
{
"long_name" : "94043",
"short_name" : "94043",
"types" : [ "postal_code" ]
}
],
"formatted_address" : "1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA",
"geometry" : {
"location" : {
"lat" : 37.42291810,
"lng" : -122.08542120
},
"location_type" : "ROOFTOP",
"viewport" : {
"northeast" : {
"lat" : 37.42426708029149,
"lng" : -122.0840722197085
},
"southwest" : {
"lat" : 37.42156911970850,
"lng" : -122.0867701802915
}
}
},
"types" : [ "street_address" ]
}
],
"status" : "OK"
}
Here is some sample code that will help you to grab the latitude and longitude :
public static void main(String[] args) {
try
{
URL url = new URL("http://maps.googleapis.com/maps/api/geocode/json?address=1600+Amphitheatre+Parkway,+Mountain+View,+CA&sensor=false");
URLConnection conn = url.openConnection();
conn.connect();
InputStreamReader isr = new InputStreamReader(conn.getInputStream());
StringBuffer sbLocation = new StringBuffer();
for (int i=0; i != -1; i = isr.read())
{
sbLocation.append((char)i);
}
String getContent = sbLocation.toString().trim();
if(getContent.contains("results"))
{
String temp = getContent.substring(getContent.indexOf("["));
JSONArray JSONArrayForAll = new JSONArray(temp);
String lng = JSONArrayForAll.getJSONObject(0).getJSONObject("geometry").getJSONObject("location").get("lng").toString();
String lat = JSONArrayForAll.getJSONObject(0).getJSONObject("geometry").getJSONObject("location").get("lat").toString();
System.out.println(" Latitude : " + lat);
System.out.println(" Longitude : " + lng);
}
}
catch (MalformedURLException e)
{
e.printStackTrace();
}
catch (IOException e)
{
e.printStackTrace();
}
}

MongoDB JSON array within JSON object field removal

I have a json object as following:
{ "_id" : ObjectId("508806803bb97dc546e6f307"), "user_name" : "user1", "user_id" : 45645645, "likes" : [ { "event_id" : NumberLong("4578541212") },{ "event_id" : NumberLong("4578541213") } ], "dislikes" : [ ] }
I'm trying to delete specific event within likes array via java drivers
tried doing this first in shell:
> db.users.update( {'likes.event_id' : 4578541212}, { '$unset':{'likes.event_id'
:1}})
with no luck...how can I manage doing that?

If you want to just remove the event_id field from the array element:
db.users.update( {'likes.event_id' : 4578541212}, {'$unset':{'likes.$.event_id' :1}})
Use the $pull operator to delete the element:
db.users.update({'likes.event_id': 4578541212}, {'$pull':{likes: {event_id: 4578541212}}})