We Keep Coding

Get access to a key in Java HttpClient response

New to API's in Java. I'm trying to get a response from an API and print some of it.
My response in the code below prints the whole dictionary, but I want to print just part of it. In the current way, I have no idea how I can get access to the dictionary, as my response is a String and I couldn't find a relevant BodyHandlers method.
How can I do that? Thanks a lot.
This is my code:
HttpRequest urlAnalysisRequest = HttpRequest.newBuilder()
.uri(URI.create("https://www.virustotal.com/api/v3/analyses/....(I put the id here)"))
.header("Accept", "application/json")
.header("x-apikey", "....(I put api key here)")
.method("GET", HttpRequest.BodyPublishers.noBody())
.build();
HttpResponse<String> urlAnalysisResponse;
try {
urlAnalysisResponse = HttpClient.newHttpClient().send(urlAnalysisRequest, HttpResponse.BodyHandlers.ofString());
System.out.println(urlAnalysisResponse.body());
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
This is my response. I'm trying to get "stats":
{
"meta": {
"url_info": {
"url": "http://www.facebook.com/",
"id": "114fb86b9b4e868f8bac2249eb5c444b545f0240c3dadd23312a0bc1622b5488"
}
},
"data": {
"attributes": {
"date": 1641238171,
"status": "completed",
"stats": {
"harmless": 84,
"malicious": 0,
"suspicious": 0,
"undetected": 9,
"timeout": 0
},
..........

The response is written in a format known as JSON. You need a library to parse this. There are a few options; I strongly suggest you go with Jackson.
You can choose to refer to it with string paths, or, you can make a java class that 'matches' this output, e.g:
class VirusTotalResponse {
VTMeta meta;
VTData data;
}
class VTMeta {
VTUrlInfo urlInfo;
}
class VTUrlInfo {
String url;
String id;
}
and so on. With all those classes in place, turn them all into proper classes (use your IDE's various options, or use Project Lombok) and then just ask Jackson to turn that response into an instance of VirusTotalResponse and you'll have a nice shiny java object, you can then just:
int harmless = response.getData().getAttributes().getStats().getHarmless();

You can use ObjectMapper which is library for handling JSON data easily. You can use not only Class for mapping, but also Map.
I recommend this tutorial site.
https://www.baeldung.com/jackson-object-mapper-tutorial

Ninja framework endpoint throws 500 error when trying to map JSON to custom object

So I've got a Ninja endpoint here:
public Result processRecurring(Context context, RecurOrderJSON recurOrderJSON) {
String id = recurOrderJSON.id;
String event_type = recurOrderJSON.event_type;
String request_id = recurOrderJSON.request_id;
//Map data = recurOrderJSON.data;
//recurringRouter(event_type, data);
log.info("ID value");
log.info(id);
return JsonResponse.build()
.message("OK")
.toResult();
}
The class I am trying to map to:
public class RecurOrderJSON {
public String id;
public String event_type;
public String request_id;
// Maybe switch data type?
//public Map data;
}
And the route:
router.POST().route("/recurring").with(RecurringController::processRecurring);
I am just trying to send some simple JSON to a webhook and for some reason the object mapping doesn't seem to be working. I think maybe I am misunderstanding the documentation?
http://www.ninjaframework.org/documentation/working_with_json_jsonp.html
The example they give you is this:
If you send that JSON to your application via the HTTP body you only need to add the POJO class to the controller method and Ninja will parse the incoming JSON for you:
package controllers;
public class ApplicationController {
public Result parsePerson(Person person) {
String nameOfPerson = person.name; // will be John Johnson
...
}
}
As far as I can tell, I am doing this correctly? Am I understanding the documentation wrong? Here's an example JSON object - currently I am only trying to grab the top level strings, but I'll eventually want to grab data as well:
{
"id": "hook-XXXXX",
"event_type": "tx-pending",
"data": {
"button_id": "static",
"publisher_organization": "org-XXXXXXX",
"campaign_id": "camp-097714a40aaf8965",
"currency": "USD",
"order_currency": "USD",
"id": "tx-XXXXXXX",
"category": "new-user-order",
"modified_date": "2018-10-15T05:41:12.577Z",
"order_total": 9680,
"button_order_id": "btnorder-77c9e56fd990f127",
"publisher_customer_id": "XymEz8GO2M",
"rate_card_id": "ratecard-41480b2a6b1196a7",
"advertising_id": null,
"event_date": "2018-10-15T05:41:06Z",
"status": "pending",
"pub_ref": null,
"account_id": "acc-4b17f5a014d0de1a",
"btn_ref": "srctok-0adf9e958510b3f1",
"order_id": null,
"posting_rule_id": null,
"order_line_items": [
{
"identifier": "Antique Trading Card",
"description": "Includes Lifetime Warranty",
"amount": 9680,
"publisher_commission": 968,
"attributes": {},
"total": 9680,
"quantity": 1
}
],
"order_click_channel": "webview",
"order_purchase_date": null,
"validated_date": null,
"amount": 968,
"customer_order_id": null,
"created_date": "2018-10-15T05:41:12.577Z",
"commerce_organization": "org-XXXXXX"
},
"request_id": "attempt-XXXXXXX"
}
Currently I am just trying to get the string values, yet I am constantly getting a 500 error and no other indication in my logs of any error.
As far as I can tell, Ninja should just automatically map the JSON to my object, correct?

I successfully reproduced your issue, and then fixed it.
First, for easy way to try/test, I recommend (temporary) modifications:
package controllers;
import models.RecurOrderJSON;
import ninja.Context;
import ninja.Result;
public class RecurringController {
public Result processRecurring(Context context, RecurOrderJSON recurOrderJSON) {
log.info("recurOrderJSON => " + recurOrderJSON);
return ninja.Results.ok();
}
}
And then, update your model this way:
package models;
import java.util.Map;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
#JsonIgnoreProperties(ignoreUnknown = true)
public class RecurOrderJSON {
public String id;
public String event_type;
public String request_id;
public Map data;
#Override
public String toString() {
return "RecurOrderJSON [id=" + id + ", event_type=" + event_type + ", request_id=" + request_id + ", data="
+ data.toString() + "]";
}
}
You can notice:
The data type must stay raw (generic can't be used here)
the important #JsonIgnoreProperties(ignoreUnknown = true) annotation to avoid deserialize issue, if ever your source data does not perfectly match your model (be sure to use the recent version of annotation, in fasterxml sub-package, instead of the old one, in codehaus sub-package)
the toString() implementation only allowing quick check of OK/KO deserialization
Then you can easily test the system with wget, or curl:
curl -H 'Content-Type: application/json' -d "#/tmp/jsonINput.json" -X POST http://localhost:8080/recurring
Notice it is very important to specify the Content-type for good interpretation.
With the /tmp/jsonINput.json file containing exactly the json contents you specified in your question.
This way, everything is working like a charm, obtaining this output:
recurOrderJSON => RecurOrderJSON [id=hook-XXXXX, event_type=tx-pending, request_id=attempt-XXXXXXX, data={button_id=static, publisher_organization=org-XXXXXXX, campaign_id=camp-097714a40aaf8965, currency=USD, order_currency=USD, id=tx-XXXXXXX, category=new-user-order, modified_date=2018-10-15T05:41:12.577Z, order_total=9680, button_order_id=btnorder-77c9e56fd990f127, publisher_customer_id=XymEz8GO2M, rate_card_id=ratecard-41480b2a6b1196a7, advertising_id=null, event_date=2018-10-15T05:41:06Z, status=pending, pub_ref=null, account_id=acc-4b17f5a014d0de1a, btn_ref=srctok-0adf9e958510b3f1, order_id=null, posting_rule_id=null, order_line_items=[{identifier=Antique Trading Card, description=Includes Lifetime Warranty, amount=9680, publisher_commission=968, attributes={}, total=9680, quantity=1}], order_click_channel=webview, order_purchase_date=null, validated_date=null, amount=968, customer_order_id=null, created_date=2018-10-15T05:41:12.577Z, commerce_organization=org-XXXXXX}]

Given the specific input code with data field commented out
//public Map data;
and the posted input JSON that includes this field, the request should fail with 400 Bad Request.
The reason being that Ninja uses Jackson for JSON parsing and it will throw on unknown fields by default.
The quick workaround is to add #JsonIgnoreProperties annotation to RecurOrderJSON class.
e.g.
#JsonIgnoreProperties(ignoreUnknown = true)
public class RecurOrderJSON {
...
}
See: Ignoring new fields on JSON objects using Jackson
Now if the error was not 400 there isn't much information to go by as there doesn't seem to be anything else obviously wrong with the code.
Either post an SSCCE demonstrating the problem or attempt to debug by surfacing the error page with the following method:
Launch the application in debug mode with mvn package ninja:run
Access the end-point with a tool that allows to inspect the response in detail such as curl e.g.
Store request JSON in input.json
Run curl -v -o result.html -H 'Content-Type: application/json' --data '#input.json' http://localhost:8080/recurring
Open result.html to examine the response

Might it be that you are performing a bad request (hence the JSON is not found) but for some Ninja bug it returns error 500?
For example you can take a look here where is stated that parsing an empty JSON in a JSON request does leads to a misguiding error (500) while it is supposed to return 400 "Bad Request"

Context not needed in processRecurring and use Results.json() and return original
public Result processRecurring(RecurOrderJSON recurOrderJSON) {
String id = recurOrderJSON.id;
String event_type = recurOrderJSON.event_type;
String request_id = recurOrderJSON.request_id;
//Map data = recurOrderJSON.data;
//recurringRouter(event_type, data);
log.info("ID value");
log.info(id);
return Results.json().render(recurOrderJSON);
}
Make sure you get the namespace in your RecurOrderJSON
package models;
public class RecurOrderJSON {
public String id;
public String event_type;
public String request_id;
// Maybe switch data type?
//public Map data;
}
Good luck!

apply gson UPPER_CAMEL_CASE for specific sub-json

I have these messages arriving from SQS:
{
"eventID": "zzz",
"eventName": "MODIFY",
"eventVersion": "1.1",
"eventSource": "aws:dynamodb",
"awsRegion": "us-east-1",
"dynamodb": {
"ApproximateCreationDateTime": 1521976320,
"Keys": {
"key_1": {
"S": "yyy"
},
"key_2": {
"S": "xxx"
}
},
"SequenceNumber": "123",
"SizeBytes": 321,
"StreamViewType": "KEYS_ONLY"
},
"eventSourceARN": "arn:aws:dynamodb:us-east-1:eventSourceARN",
"itemType": "myItem"
}
I want to use gson library to convert this json string into a Record object (com.amazonaws.services.dynamodbv2.model.Record) which contains a StreamRecord object (com.amazonaws.services.dynamodbv2.model.StreamRecord) that represents the dynamodb sub json.
problem is that the inner fields of the dynamodb object are PascalCase while the other fields are normal camelCase.
This code:
Gson gson = new GsonBuilder()
//.setFieldNamingPolicy(FieldNamingPolicy.UPPER_CAMEL_CASE)
.create();
String json = <the json from the example above>
Record record = gson.fromJson(json, Record.class);
log.info("record="+record.toString());
StreamRecord dynamodb = record.getDynamodb();
log.info("dynamodb="+dynamodb.toString());
Map<String, AttributeValue> keys = dynamodb.getKeys();
log.info("keys="+keys.toString());
prints this log (UPPER_CAMEL_CASE commented out) :
record={EventID: zzz,EventName: MODIFY,EventVersion: 1.1,EventSource: aws:dynamodb,AwsRegion: us-east-1,Dynamodb: {},}
and then throws Null Pointer exception because the dynamoDb object is empty - because my json string is UPPER_CAMEL_CASE, while in the object its normal camelCase.
I want to apply FieldNamingPolicy.UPPER_CAMEL_CASE only for the dynamodb sub json.
perhaps somehow using FieldNamingStrategy ?
The json is given and I cannot change its schema.
I also can't change the fact that I get it as string.
see AWS API:
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_Record.html
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_StreamRecord.html

You seem to want the following naming strategy:
private static final Gson gson = new GsonBuilder()
.setFieldNamingStrategy(field -> {
if ( field.getDeclaringClass() == StreamRecord.class ) {
return FieldNamingPolicy.UPPER_CAMEL_CASE.translateName(field);
}
return FieldNamingPolicy.IDENTITY.translateName(field);
})
.create();
I usually never use naming strategies in favor of the #SerializedName annotation though, just to be more precise when declaring mappings.

How to use the PMML model in Java?

i have build a python file based on randomforestclassifier and made a PMML model, now we need to use this PMML in Java to classify the data into 2 catagories..but this is new to me and i don't know how to handle the java part..

Google: pmml java
Second Link is https://github.com/jpmml/jpmml-evaluator
There you have a library with examples. So try it and come back if you have problems.

Another choice is PMML4S that is implemented in Scala, but you are free to use it by Scala or Java API. It's very easy to use, for example:
import org.pmml4s.model.Model;
Model model = Model.fromFile("/the/pmml/file/path");
Object result = model.predict(data)
The data could be in Map, Array, A String in JSON, Series, the result type is same as the input. For details how to use PMML4S in Java, you could see the example: https://github.com/autodeployai/pmml4s/blob/master/src/test/java/org/pmml4s/model/JModelTest.java

There is a demo base on high performance and light framework Vert.x:
vertx-pmml
Just config router via Json like this:
"route": {
"/predict/iris": {
"access": "public",
"method": "POST",
"actor": "evaluter.predict",
"ext": {
"name": "IRIS_SVC",
"pmml": ".\\config\\model_svc.pmml",
"version": "1.0.0"
}
}
...
Then prefect your feature handler(not nescessary if feed a json map match your pmml inputfields):
//src/main/java/com/hirisklab/evaluate/evaluator/actor/EvaluateImpl.java
private Future<Map<String, Object>> Featurelize(JsonObject data) {
Promise<Map<String, Object>> promise = Promise.promise();
try {
Map<String, Object> feature = data.getMap();
// TODO: featurelize with pmml input fields...
promise.complete(feature);
} catch (Exception e) {
promise.fail(e);
}
return promise.future();
}
The action assigned in router(actor field) will handle at here:
//src/main/java/com/hirisklab/evaluate/evaluator/actor/EvaluateImpl.java
public void predict(JsonObject data, Handler<AsyncResult<EvaluateResponse<JsonObject>>> handler) {
Promise<EvaluateResponse<JsonObject>> promise = Promise.promise();
try {
JsonObject profile = Optional.ofNullable(data.getJsonObject("_EXT")).orElseThrow(() -> EvaluateException.InvalidConfigException);
EvaluaterFactory.getEvaluater(profile).onSuccess(evaluater -> {
Featurelize(data.getJsonObject("data")).onSuccess(feature -> {
evaluater.predict(feature).onSuccess(result -> {
promise.complete(new EvaluateResponse<JsonObject>(new JsonObject().put("raw", result)));
}).onFailure(f -> promise.fail(EvaluateException.FailedToPredict));
}).onFailure(f -> promise.fail(EvaluateException.FailedToFeaturelize));
}).onFailure(f -> promise.fail(EvaluateException.FailedToLoadEvaluator));
} catch (Exception e) {
e.printStackTrace();
promise.fail(e);
}
handler.handle(promise.future());
}

Here is the simple code that might help you to get the direction:
// This will load the PMML file
Evaluator evaluator = new LoadingModelEvaluatorBuilder()
.load(new File("path\\file.pmml"))
.build();
// The internal self check
evaluator.verify();
System.out.println("PMML Loaded");
// This will create the `actual` pipeline from the PMML and load in java
TransformerBuilder pmmlTransformerBuilder = new TransformerBuilder(evaluator)
.withTargetCols()
.withOutputCols()
.exploded(true);
System.out.println("Building in...");
Transformer pmmlTransformer = pmmlTransformerBuilder.build();
// Now we are loading the file and converting that into the data frame so that
// we can use it to transform into the prediction into the PMML model
Dataset<Row> DF = sparkSession.read()
.option("header", true)
.option("inferSchema", true)
.csv("path\\file.csv");
DF.printSchema();
// This will predict the new data points from the pipeline
Dataset<Row> result = pmmlTransformer.transform(DF);
NOTE: Make sure, you are loading the correct csv file there. There should not be any change in the column headers otherwise it will show an error.
Here is the link: JPMML-Evaluator (the same which is provided by Florian) to explore more about this.

JSON validation using JSONAssert with regular expressions

I am working in a open-source project which uses REST interface. To validate (match actual response with expected) our rest interfaces in the JUnit, we would like to use the JSONAssert. (https://github.com/skyscreamer/JSONassert). But I have a problem with the usage.Kindly help to resolve it.
Expected JSON:
{
"objectId": "[^\\s]",
"protocol": "[^\\s]",
"hostName": "[^\\s]",
"port": "[^\\s]",
"commParams": "[^\\s]"
}
Remarks: objectId/protocol/hostName/port/commParams can be anything but should not be empty
Actual JSON:
{
"objectId": "controller2",
"protocol": "ftp",
"hostName": "sdnorchestrator",
"port": "21",
"commParams": "username:tomcat, password:tomdog"
}
Problem1: Which interface of JSON Assert, i need to use to solve the above issue:Below one?
JSONAssert.assertEquals("Expected JSON", "Actual JSON" new CustomComparator(
JSONCompareMode.LENIENT_ORDER, new Customization(PREFIX, new RegularExpressionValueMatcher<Object>())));
Problem 2: What should be the PREFIX here?(I tried with "", "., "." but had no success)
Any other recommendation (other than JSONAssert) for the above problem is also welcome.

If you want to globally apply regular expression customizations with JSONAssert, you can construct a single Customization with path = "***", and use the RegularExpressionValueMatcher constructor with no arguments.
Example:
final String expectedJson = "{objectId: \"[^\\s]+\", protocol: \"[^\\s]+\"}";
final String actualJson = "{\"objectId\": \"controller2\", \"protocol\": \"ftp\"}";
JSONAssert.assertEquals(
expectedJson,
actualJson,
new CustomComparator(
JSONCompareMode.LENIENT,
new Customization("***", new RegularExpressionValueMatcher<>())
)
);
This assertion passes successfully (tested with JSONassert version 1.5.0).

I have found better alternative.Usage of JsonSchema validator would solve most of the problem (http://json-schema.org/). Write the json schema in expected response and validate it using json validator jar. (json-schema/json-schema-validator-2.0.1.jar.zip( 166 k))

I could not get the path to except an RE either. Eventually I just over-rode the compareValues of DefaultComparator to force it to apply the RegularExpressionValueMatcher to all paths:
import org.json.JSONException;
import org.skyscreamer.jsonassert.JSONAssert;
import org.skyscreamer.jsonassert.JSONCompareResult;
import org.skyscreamer.jsonassert.RegularExpressionValueMatcher;
import org.skyscreamer.jsonassert.ValueMatcherException;
import org.skyscreamer.jsonassert.comparator.DefaultComparator;
import static org.skyscreamer.jsonassert.JSONCompareMode.STRICT;
...
final RegularExpressionValueMatcher reMatcher = new RegularExpressionValueMatcher();
JSONAssert.assertEquals( "{\"key\":\"v.*\"}", "{\"key\":\"value\"}",
new DefaultComparator( STRICT ) {
#Override
public void compareValues(String prefix, Object expectedValue, Object actualValue, JSONCompareResult result) throws JSONException {
try {
if( !reMatcher.equal(actualValue, expectedValue) ) result.fail(prefix, expectedValue, actualValue);
} catch( ValueMatcherException e ) { result.fail(prefix, e); }
}
}