We Keep Coding

How to properly store event.duration field in Elastic?

I have a Java service that writes logs in JSON format, they are then picked up by filebeat and sent to Elastic. I would like to be able to set one of the ECS fields (event.duration) described here
I set up a net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder encoder, and I set the event.duration field in MDC before calling logging method. The output looks like this:
{
"#timestamp": "2021-12-07T10:41:59.589+01:00",
"message": "message",
"event.duration": "5606000000",
"service": {
"name": "logging.application.name_IS_UNDEFINED",
"type": "java"
},
"log": {
"logger": "com.demo.Demo",
"level": "WARN"
},
"process": {
"thread": {
"name": "main"
}
},
"error": {}
}
However, in Kibana I see event.duration as a JSON inside the flat field:
flat
{
"event.duration": "10051000000"
}
How can I make it on the same level as other ECS fields like event.name?

You should create an ingest pipeline using the dot_expander processor in order to transform your dotted field into an object:
PUT _inest/pipeline/de-dot
{
"processors" : [
{
"dot_expander": {
"field": "event.duration"
}
}
]
}
Then you need to make sure that your indexing process references this pipeline, i.e. ...?pipeline=de-dot

Getting Extra information while wrapping List<Objects> in http JAX RS response

{
"context": {
"headers": {},
"entity": {
"validationDetailsEntityList": [
{
"createTimestamp": 1512653225936,
"modifyTimestamp": 1512653225936,
"version": 0,
"auditTimestamp": "2"
},
{
"createTimestamp": 1512652876650,
"modifyTimestamp": 1512652876650,
"version": 0,
"auditTimestamp": "2"
},
{
"createTimestamp": 1512652955832,
"modifyTimestamp": 1512652955832,
"version": 0,
"auditTimestamp": "2"
}
]
}
"entityType":"com.example.demo.wrapper.ABCDWrapper",
"entityAnnotations": [],
Class Written below to get the response on the http mapped request
#RequestMapping(value = "/fetch", method = RequestMethod.POST)
public Response getAllXYZDetails(#RequestBody QueryDetails queryDetailsPayLoad) {
List<XYZEntity> xyzEntityList = xyzService.getAllXYZDetails();
return Response.ok(xyzEntityList)
.build();
}
I am trying to build a generic response type from my controller class on http REST calls, and so my return type is Response.
Now, what's happening is that:
The response generated not only has the details that I want in json but also it is having a lot of extra information like
entityType and entityAnnotations etc etc(SEE ABOVE RESPONSE), which I don't want.
How to get rid of those ans get only the entities in response?

If you are using Jackson annotations you can configure pretty much everything. In your case, #JsonIgnore should suffice.
If you are using JAX-RS/JAXB then Add #JsonIgnoreProperties(ignoreUnknown = true) on top of your class

Handle "Date": { "#nil": "true"} during jackson parsing

I am trying to parse json using jackson parser. Jackson parse the data successfully, but when #nil:true comes in the json, code breaks and gives an error:
Can not deserialize instance of java.lang.String out of START_OBJECT token
json string like this:
[
{
"Users": {
"UserID": "1",
"ExpirationDate": {
"#nil": "true"
}
}
}
]
but when json comes like below, then parsing happens successfully:
[
{
"Users": {
"UserID": "1",
"ExpirationDate": "2016-07-07"
}
}
}
]
Can you please let me know, how to handle such type of cases.

swagger io consume json on post

Using swagger editor I created a post call to consume a json object that I want to simply load into a db but when I run the call I get an empty json object.
This is the parameters portion of my json swagger for the post
"parameters": [
{
"in": "body",
"name": "body",
"description": "Add question to collection",
"required": true,
"schema": { "type" : "object", "additionalProperties" : {}
}
}
],
It then creates a "Body" model, but I am not able to see the json that was part of the post:
#javax.annotation.Generated(value = "class io.swagger.codegen.languages.JaxRSServerCodegen", date = "2016-01-22T20:49:03.229Z")
public class Body {
#Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
Body body = (Body) o;
return true;
}
#Override
public int hashCode() {
return Objects.hash();
}
#Override
public String toString() {
StringBuilder sb = new StringBuilder();
sb.append("class Body {\n");
sb.append("}");
return sb.toString();
}
/**
* Convert the given object to string with each line indented by 4 spaces
* (except the first line).
*/
private String toIndentedString(Object o) {
if (o == null) {
return "null";
}
return o.toString().replace("\n", "\n ");
}
}
If I remove the text/json from consume and generated my code again and I still see an issue with the body model and being able to pull in json.
If you look at the toString method it shows hard coded values, so I dont see how I can pull the json from the post with the post method only taking in the Body and securitycontext.

I am a little confused about the http accept, when using this swagger snippet:
"post": {
"tags": [
"AskGrey"
],
"summary": "Create new askgrey.com question",
"operationId": "postAskGrey",
"consumes": [
"application/json",
"text/json"
],
"produces": [
"application/json"
],
"parameters": [
{
"in": "body",
"name": "body",
"description": "Add question to collection",
"required": true,
"schema": { "type" : "object", "additionalProperties" : {}
}
}
],
"responses": {
"200": {
"description": "OK",
"schema": {
"type": "object"
}
}
},
"deprecated": false
}
The method generated is the following:
#Override
#POST
#Consumes("application/json")
public Response postAskGrey(Body body,SecurityContext securityContext)
throws NotFoundException {
So based on all this I am not sure how to pull in the post body info, normally I would grab what I need from the http request but with swagger I cant seem to figure out how that gets used.

Before you send your data you should check HTTP ACCEPT method which you set in swagger when you send your data.
There should be Several Accept method which their behaviour distinct from each other when sending data to server.
Thus for application/JSON : Data are part of body.
form-data and x-www-form-urlencoded : Data are part of header.
I haven't adaquate Java experience to give correct code to obtain json into related jSON Object but How to convert HTTP Request Body into JSON Object in Java this answer could help.
Please Check following RFCs for further information
form-data related RFC https://www.rfc-editor.org/rfc/rfc7578
x-www-form-urlencoded related RFC https://datatracker.ietf.org/doc/html/draft-hoehrmann-urlencoded-01
application/JSON related rfc https://www.ietf.org/rfc/rfc4627.txt
UPDATED
Related curl command : I get the command from swagger live demo http://petstore.swagger.io/#/pet paste your json to it and change url, secret key give it a try!
curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
"id": 0,
"category": {
"id": 0,
"name": "string"
},
"name": "doggie",
"photoUrls": [
"string"
],
"tags": [
{
"id": 0,
"name": "string"
}
],
"status": "available"
}' 'http://petstore.swagger.io/v2/pet?api_key=special-key'

Validate JSON to a Particular Format

I Have a Json which may come from other application and i need to check it whether is is in particular format. The JSON template i have is as follows,
{
"Types": {
"Type1": {
"attribute1": "value1",
"attribute2": "value2",
"attribute3": "value3",
"recordList": {
"record1": [
{
"field": "value"
},
{
"field": {
"subrecord1": [
{
"subfield1": "subvalue1",
"subfield2": "subvalue2"
}
]
}
}
]
},
"date": "2010-08-21 03:05:03"
}
}
}
Is there any way to validate the JSON based on particular template or format.

You can use JSON Schema for that. JSON Schema lets you describe the format of the object graph you expect to receive, and then software implementing it lets you validate what you receive against your schema. There's an OSS Java implementation called json-schema-validator.