We Keep Coding

Elasticsearch Java API from 2.x to 5.x issues

I've updated to elasticsearch java library version 5.2.0.
In 2.x,
I was using SearchRequestBuilder.addField() in order to add a field to the search request. Nevertheless, It seems to be replaced. I've written the available methods intellisense is showing me. Which of them do I need to pick?
addDocValueField
addFieldDataField
addScriptField
addStoredField
storedFields
fields
SearchRequestBuilder.setNoFields is also removed. Which would be the alternative?
Currently, I'm calling scripts from Java using this code. Is there any more elegant way to call it in 5.x Java API?
Code:
return AggregationBuilders
.terms(this.getName())
.field(this.getName())
.script(new Script(
ScriptType.FILE,
"painless",
"year",
ImmutableMap.of("field", this.getName())
)
);
As you can see I setting field as script parameter. Nevertheless, I don't quite understand how to get it from script code.
Thanks.

When in doubt, go to the source
use setFetchSource(String[] includes, String[] excludes) instead
use setFetchSource(false) instead
if you need to execute this script for each document, you can use addScriptField()

How to create a tensorflow serving client for the 'wide and deep' model?

I've created a model based on the 'wide and deep' example (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/learn/wide_n_deep_tutorial.py).
I've exported the model as follows:
m = build_estimator(model_dir)
m.fit(input_fn=lambda: input_fn(df_train, True), steps=FLAGS.train_steps)
results = m.evaluate(input_fn=lambda: input_fn(df_test, True), steps=1)
print('Model statistics:')
for key in sorted(results):
print("%s: %s" % (key, results[key]))
print('Done training!!!')
# Export model
export_path = sys.argv[-1]
print('Exporting trained model to %s' % export_path)
m.export(
export_path,
input_fn=serving_input_fn,
use_deprecated_input_fn=False,
input_feature_key=INPUT_FEATURE_KEY
My question is, how do I create a client to make predictions from this exported model? Also, have I exported the model correctly?
Ultimately I need to be able do this in Java too. I suspect I can do this by creating Java classes from proto files using gRPC.
Documentation is very sketchy, hence why I am asking on here.
Many thanks!

I wrote a simple tutorial Exporting and Serving a TensorFlow Wide & Deep Model.
TL;DR
To export an estimator there are four steps:
Define features for export as a list of all features used during estimator initialization.
Create a feature config using create_feature_spec_for_parsing.
Build a serving_input_fn suitable for use in serving using input_fn_utils.build_parsing_serving_input_fn.
Export the model using export_savedmodel().
To run a client script properly you need to do three following steps:
Create and place your script somewhere in the /serving/ folder, e.g. /serving/tensorflow_serving/example/
Create or modify corresponding BUILD file by adding a py_binary.
Build and run a model server, e.g. tensorflow_model_server.
Create, build and run a client that sends a tf.Example to our tensorflow_model_server for the inference.
For more details look at the tutorial itself.

Just spent a solid week figuring this out. First off, m.export is going to deprecated in a couple weeks, so instead of that block, use: m.export_savedmodel(export_path, input_fn=serving_input_fn).
Which means you then have to define serving_input_fn(), which of course is supposed to have a different signature than the input_fn() defined in the wide and deep tutorial. Namely, moving forward, I guess it's recommended that input_fn()-type things are supposed to return an InputFnOps object, defined here.
Here's how I figured out how to make that work:
from tensorflow.contrib.learn.python.learn.utils import input_fn_utils
from tensorflow.python.ops import array_ops
from tensorflow.python.framework import dtypes
def serving_input_fn():
features, labels = input_fn()
features["examples"] = tf.placeholder(tf.string)
serialized_tf_example = array_ops.placeholder(dtype=dtypes.string,
shape=[None],
name='input_example_tensor')
inputs = {'examples': serialized_tf_example}
labels = None # these are not known in serving!
return input_fn_utils.InputFnOps(features, labels, inputs)
This is probably not 100% idiomatic, but I'm pretty sure it works. For now.

Extract details from syslog message with java grok patterns

I am working on a Java project to parse logs where I am using Java Grok library for pattern recognition. I have given the pattern as follows:
%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}
When I try parsing the line,
Dec 23 14:30:01 louis CRON[619]: (www-data) CMD (php /usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log)
it gives the following output:
syslog_timestamp=Dec 23 14:30:01
syslog_hostname=louis
syslog_program=CRON
syslog_pid=619
syslog_message=(www-data) CMD (php /usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log)
I want to extract details from the syslog_message further to get stuff like facility and severity. How can I improve the grok pattern to get these details?

I am not an expert of Java Grok. You can use www-data as your facility and error as your severity.
Change your pattern to get this information.
syslog_facility=www-data
syslog-severity=error
Regards,
Ivo

As far as I see, you can't get the severity and facility from that message because it's not there. Assuming this is written by a syslog daemon, and assuming that syslog daemon is rsyslog, you can change the way things are written down by altering the template and select the properties you want.
You can write, for example, one JSON per line in a file with a template like this (basically copy-pasted from this rsyslog->Elasticsearch blog post):
template(name="jsonPerLine"
type="list") {
constant(value="{")
constant(value="\"#timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\"}\n")
}
This way, in your application you can parse this JSON and get all these fields, which is much cheaper than using regular expressions (which is what grok does).
You'd use this template to write to files later on in the configuration, for example:
action(
type="omfile"
file="/var/log/json_messages"
template="jsonPerLine"
)
If you get errors from the syntax above, it may be that your distro has a really-really old version of rsyslog. If that's so, you can get packages from the official repositories.

Consuming SOAP Webservice - Java VS PHP

a general question:
We are launching a new ITSM Toolsuite in our company called ServiceNow.
ServiceNow offers a lot of nice out-of-the-box Webservices.
Currenty we are implementing some interfaces to other interal systems and we use these Webservices to consume data of Servicenow.
How we did it in PHP:
<?php
$credentials = array('login'=>'user', 'password'=>'pass');
$client = new SoapClient("https://blah.com/incident.do?WSDL", $credentials);
$params = array('param1' => 'value1', 'param1' => 'value1');
$result = $client->__soapCall('getRecords', array('parameters' => $params));
// result array stored in $result->getRecordsResult
?>
And thats it! 5 minutes of work, Beautiful and simple - from my point of view.
Ok and now the same in Java:
I did some research and it seems everbody is using Apache Axis2 for consuming Webservices in Java. So I decided to go down that road.
Install Apache Axis
open cygwin or cmd and generate Classes from WSDL.. WTF? what for?
$ ./wsdl2java.sh -uri https://blah.com/incident.do?WSDL
copy generated classes to Java Project in Eclipse.
Use this classes:
ServiceNow_incidentStub proxy = new ServiceNow_incidentStub();
proxy._getServiceClient().getOptions().setProperty(org.apache.axis2.transport.http.HTTPConstants.CHUNKED, Boolean.FALSE);
ServiceNow_incidentStub.GetRecords defectsGetRecords = new ServiceNow_incidentStub.GetRecords();
ServiceNow_incidentStub.GetRecordsResponse defectsResult = new ServiceNow_incidentStub.GetRecordsResponse();
proxy._getServiceClient().getOptions().setManageSession(true);
HttpTransportProperties.Authenticator basicAuthentication = new HttpTransportProperties.Authenticator();
basicAuthentication.setUsername("user");
basicAuthentication.setPassword("pass");
proxy._getServiceClient().getOptions().setProperty(org.apache.axis2.transport.http.HTTPConstants.AUTHENTICATE, basicAuthentication);
defectsResult = proxy.getRecords(defectsGetRecords);
com.service_now.www.ServiceNow_incidentStub.GetRecordsResult_type0[] defects = defectsResult.getGetRecordsResult();
for (int j=0; j < defects.length; j++) {
// do something
}
Its working but I think this way is very complicated..
everytime something in the wsdl changes - i must recompile them with axis.
There is no way to configure something globally like Soap-endpoint or something like that.
Is there an easier way in Java to consume SOAP with a WSDL??

First off: I completely agree. I do quite a bit of work with Web Services and ServiceNow, and using Java and/Or .Net is quite different than using a scripted language (I usually use Perl for scripts). The inherent issue comes into the fact that a WSDL should not be changing that often, especially in production. The idea in Java and .Net is that you get these stub classes to get compile time error checking.
If your currently in a Ph1 and haven't deployed Prod yet, then you should really look into how often that WSDL will be changing. Then make your decision from there on which technology to use. The nice thing is that even if the WSDL changes, posting data to the instance - almost all of the fields are optional. So if a new field is added it's not a big deal. The issue comes in when data is returned (most of the time) because many times java and .net will throw an exception if the returned XML is not in the structure it is expecting.
One thing that many people do is setup Modules as CI's in the CMDB and maintain their ServiceNow instance through the Change Request module. That way your java application will be a downstream CI to whatever module/table you are querying, and when a CR is put in to modify that table, it will be known immediately that there will be an impact on your internal application as well.
Unfortunately you are right though, that is a trade off with the different languages and from my experience there is very little we can do to change that.
One thing I forgot to add, another option for you is to use the JSON service instead. That will allow you to make raw requests to the SNC instance then use a JSON parser to parse that data for you "on the fly" so to speak. It takes away the compile time checking but also takes away many of the flaws of the SOAP system.

IF you are using maven, try using this plugin.
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>axistools-maven-plugin</artifactId>
<version>1.4</version>
<configuration>
<urls>
<url>https://blah.com/incident.do?WSDL</url>
</urls>
<packageSpace>your.destination.package</packageSpace>
<serverSide>true</serverSide>
<outputDirectory>src/main/java</outputDirectory>
</configuration>
<executions>
<execution>
<goals><goal>wsdl2java</goal></goals>
</execution>
</executions>
</plugin>

I was also trying to access ServiceNow from Java using Eclipse, and it seemed to me that the Axis2 approach was overly restrictive given how ServiceNow designed their API, so I wrote my own package to generate SOAP calls dynamically using JDOM. Here is an example of what the code looks like:
Instance instance = new Instance("https://blah.service-now.com", "username", "password");
GlideFilter filter = new GlideFilter("category=network^active=true");
GlideRecordIterator iter = instance.table("incident").
bulkFetcher().setFilter(filter).getAllRecords().iterator();
while (iter.hasNext()) {
GlideRecord rec = iter.next();
System.out.println(
rec.getField("number") + " " + rec.getField("short_description"));
}
A couple of things about this code:
I use run-time validation rather than build-time validation. If you mistakenly type getField("shortdescription") the code throws an InvalidFieldNameException.
Queries are not bound by ServiceNow's normal 250 record limit because the BulkFetcher loops internally making as many Web Service calls as necessary to retrieve all the data.
The package source code is at https://sourceforge.net/projects/servicenowpump/

I consume lots of Soap services with PHP in the company I work for, and I would always suggest generating classes for the request and response data structure. Otherwise you will easily get lost - PHP does not preserve any remains of the original XML structure, it will all be converted to arrays and stdClass objects.
Getting classes created from the WSDL description is not that easy in PHP, as there are only a few scripts that do this - and they all have their shortcomings when it comes to WSDL files that make use of the more obscure parts of the SOAP standard. After that, you somehow have to make these classes available to your PHP script. If this is hard for you, it is a sign of a not too well organized code base. With the autoloading feature it works like a charm.
But yes, this step is entirely optional for PHP. If using only one Soap service, it'll probably make no difference.

Generate Java class from JSON?

In a Java Maven project, how do you generate java source files from JSON? For example we have
{
"firstName": "John",
"lastName": "Smith",
"address": {
"streetAddress": "21 2nd Street",
"city": "New York"
}
}
When we run mvn generate-sources we want it to generate something like this:
class Address {
JSONObject mInternalJSONObject;
Address (JSONObject json){
mInternalJSONObject = json;
}
String getStreetAddress () {
return mInternalJSONObject.getString("streetAddress");
}
String getCity (){
return mInternalJSONObject.getString("city");
}
}
class Person {
JSONObject mInternalJSONObject;
Person (JSONObject json){
mInternalJSONObject = json;
}
String getFirstName () {
return mInternalJSONObject.getString("firstName");
}
String getLastName (){
return mInternalJSONObject.getString("lastName");
}
Address getAddress (){
return Address(mInternalJSONObject.getString("address"));
}
}
As a Java developer, what lines of XML do I need to write in my pom.xml in order to make this happen?

Try http://www.jsonschema2pojo.org
Or the jsonschema2pojo plug-in for Maven:
<plugin>
<groupId>org.jsonschema2pojo</groupId>
<artifactId>jsonschema2pojo-maven-plugin</artifactId>
<version>1.0.2</version>
<configuration>
<sourceDirectory>${basedir}/src/main/resources/schemas</sourceDirectory>
<targetPackage>com.myproject.jsonschemas</targetPackage>
<sourceType>json</sourceType>
</configuration>
<executions>
<execution>
<goals>
<goal>generate</goal>
</goals>
</execution>
</executions>
</plugin>
The <sourceType>json</sourceType> covers the case where the sources are json (like the OP). If you have actual json schemas, remove this line.
Updated in 2014: Two things have happened since Dec '09 when this question was asked:
The JSON Schema spec has moved on a lot. It's still in draft (not finalised) but it's close to completion and is now a viable tool specifying your structural rules
I've recently started a new open source project specifically intended to solve your problem: jsonschema2pojo. The jsonschema2pojo tool takes a json schema document and generates DTO-style Java classes (in the form of .java source files). The project is not yet mature but already provides coverage of the most useful parts of json schema. I'm looking for more feedback from users to help drive the development. Right now you can use the tool from the command line or as a Maven plugin.

If you're using Jackson (the most popular library there), try
https://github.com/astav/JsonToJava
Its open source (last updated on Jun 7, 2013 as of year 2021) and anyone should be able to contribute.
Summary
A JsonToJava source class file generator that deduces the schema based on supplied sample json data and generates the necessary java data structures.
It encourages teams to think in Json first, before writing actual code.
Features
Can generate classes for an arbitrarily complex hierarchy (recursively)
Can read your existing Java classes and if it can deserialize into those structures, will do so
Will prompt for user input when ambiguous cases exist

Here's an online tool that will take JSON, including nested objects or nested arrays of objects and generate a Java source with Jackson annotations.

Answering this old question with recent project ;-).
At the moment the best solution is probably JsonSchema2Pojo :
It does the job from the seldom used Json Schema but also with plain Json. It provides Ant and Maven plugin and an online test application can give you an idea of the tool. I put a Json Tweet and generated all the containing class (Tweet, User, Location, etc..).
We'll use it on Agorava project to generate Social Media mapping and follow the contant evolution in their API.

Thanks all who attempted to help. For me this script was helpful. It process only flat JSON and don't take care of types, but automate some routine
String str =
"{"
+ "'title': 'Computing and Information systems',"
+ "'id' : 1,"
+ "'children' : 'true',"
+ "'groups' : [{"
+ "'title' : 'Level one CIS',"
+ "'id' : 2,"
+ "'children' : 'true',"
+ "'groups' : [{"
+ "'title' : 'Intro To Computing and Internet',"
+ "'id' : 3,"
+ "'children': 'false',"
+ "'groups':[]"
+ "}]"
+ "}]"
+ "}";
JSONObject json = new JSONObject(str);
Iterator<String> iterator = json.keys();
System.out.println("Fields:");
while (iterator.hasNext() ){
System.out.println(String.format("public String %s;", iterator.next()));
}
System.out.println("public void Parse (String str){");
System.out.println("JSONObject json = new JSONObject(str);");
iterator = json.keys();
while (iterator.hasNext() ){
String key = iterator.next();
System.out.println(String.format("this.%s = json.getString(\"%s\");",key,key ));
System.out.println("}");

I'm aware this is an old question, but I stumbled across it while trying to find an answer myself.
The answer that mentions the online json-pojo generator (jsongen) is good, but I needed something I could run on the command line and tweak more.
So I wrote a very hacky ruby script to take a sample JSON file and generate POJOs from it. It has a number of limitations (for example, it doesn't deal with fields that match java reserved keywords) but it does enough for many cases.
The code generated, by default, annotates for use with Jackson, but this can be turned off with a switch.
You can find the code on github: https://github.com/wotifgroup/json2pojo

I created a github project Json2Java that does this.
https://github.com/inder123/json2java
Json2Java provides customizations such as renaming fields, and creating inheritance hierarchies.
I have used the tool to create some relatively complex APIs:
Gracenote's TMS API: https://github.com/inder123/gracenote-java-api
Google Maps Geocoding API: https://github.com/inder123/geocoding

I had the same problem so i decided to start writing a small tool to help me with this. Im gonna share andopen source it.
https://github.com/BrunoAlexandreMendesMartins/CleverModels
It supports, JAVA, C# & Objective-c from JSON .
Feel free to contribute!

I know there are many answers but of all these I found this one most useful for me. This link below gives you all the POJO classes in a separate file rather than one huge class that some of the mentioned websites do:
https://json2csharp.com/json-to-pojo
It has other converters too. Also, it works online without a limitation in size. My JSON is huge and it worked nicely.

As far as I know there is no such tool. Yet.
The main reason is, I suspect, that unlike with XML (which has XML Schema, and then tools like 'xjc' to do what you ask, between XML and POJO definitions), there is no fully features schema language. There is JSON Schema, but it has very little support for actual type definitions (focuses on JSON structures), so it would be tricky to generate Java classes. But probably still possible, esp. if some naming conventions were defined and used to support generation.
However: this is something that has been fairly frequently requested (on mailing lists of JSON tool projects I follow), so I think that someone will write such a tool in near future.
So I don't think it is a bad idea per se (also: it is not a good idea for all use cases, depends on what you want to do ).

You could also try GSON library. Its quite powerful it can create JSON from collections, custom objects and works also vice versa. Its released under Apache Licence 2.0 so you can use it also commercially.
http://code.google.com/p/google-gson/

Try my solution
http://htmlpreview.github.io/?https://raw.githubusercontent.com/foobnix/android-universal-utils/master/json/generator.html
{
"auctionHouse": "sample string 1",
"bidDate": "2014-05-30T08:20:38.5426521-04:00 ",
"bidPrice": 3,
"bidPrice1": 3.1,
"isYear":true
}
Result Java Class
private String auctionHouse;
private Date bidDate;
private int bidPrice;
private double bidPrice1;
private boolean isYear;
JSONObject get
auctionHouse = obj.getString("auctionHouse");
bidDate = obj.opt("bidDate");
bidPrice = obj.getInt("bidPrice");
bidPrice1 = obj.getDouble("bidPrice1");
isYear = obj.getBoolean("isYear");
JSONObject put
obj.put("auctionHouse",auctionHouse);
obj.put("bidDate",bidDate);
obj.put("bidPrice",bidPrice);
obj.put("bidPrice1",bidPrice1);
obj.put("isYear",isYear);

To add to #japher's post. If you are not particularly tied to JSON, Protocol Buffers is worth checking out.