We Keep Coding

XML data display in grid using kibana and logstash

I wanted to display XML data using logstash and Kibana in grid format. using below conf file I am able to display data into grid but not able to split row data.
Example:
Output
logstash.conf file :
input {
file {
path => "C:/ELK Stack/logstash-8.2.0-windows-x86_64/logstash-8.2.0/Test.xml"
start_position => "beginning"
sincedb_path => "NUL"
codec => multiline {
pattern => "^<?stations.*>"
negate => "true"
what => "previous"
auto_flush_interval => 1
max_lines => 3000
}}}
filter
{
xml
{
source => "message"
target => "parsed"
store_xml => "false"
xpath => [
"/stations/station/id/text()", "station_id",
"/stations/station/name/text()", "station_name"
]
}
mutate {
remove_field => [ "message"]
}
}
output {
elasticsearch {
action => "index"
hosts => "localhost:9200"
index => "logstash_index123xml"
workers => 1
}
stdout {
codec => rubydebug
}
}

xpath will always return arrays, to associate the members of the two arrays you are going to need to use a ruby filter. To get multiple events you can use a split filter to split an array which you build in the ruby filter. If you start with
<stations>
<station>
<id>1</id>
<name>a</name>
<id>2</id>
<name>b</name>
</station>
</stations>
then if you use
xml {
source => "message"
store_xml => "false"
xpath => {
"/stations/station/id/text()" => "[#metadata][station_id]"
"/stations/station/name/text()" => "[#metadata][station_name]"
}
remove_field => [ "message" ]
}
ruby {
code => '
ids = event.get("[#metadata][station_id]")
names = event.get("[#metadata][station_name]")
if ids.is_a? Array and names.is_a? Array y and ids.length == names.length
a = []
ids.each_index { |x|
a << { "station_name" => names[x], "station_id" => ids[x] }
}
event.set("[#metadata][theData]", a)
end
'
}
if [#metadata][theData] {
split {
field => "[#metadata][theData]"
add_field => {
"station_name" => "%{[#metadata][theData][station_name]}"
"station_id" => "%{[#metadata][theData][station_id]}"
}
}
}
You will get two events
{
"station_name" => "a",
"station_id" => "1",
...
}
{
"station_name" => "b",
"station_id" => "2",
...
}

How to get Wildfly server status via CLI?

I've been trying to validate if our server has started in Wildfly using the jboss-cli.bat
This is the command i'm using:
/host=slave-1/server-config=REST-server-one:read-resource(include-runtime=true)
and this is what i'm getting from the command
{
"outcome" => "success",
"result" => {
"auto-start" => true,
"cpu-affinity" => undefined,
"group" => "wildfly-server-group",
"name" => "wildfly-server",
"priority" => undefined,
"socket-binding-default-interface" => undefined,
"socket-binding-group" => undefined,
"socket-binding-port-offset" => 0,
"status" => "STARTED",
"update-auto-start-with-server-status" => false,
"interface" => undefined,
"jvm" => undefined,
"path" => undefined,
"ssl" => undefined,
"system-property" => undefined
}
Is there a command that will return the value of the status in that response?

You should be able to use the read-attribute operation.
/host=slave-1/server-config=REST-server-one:read-attribute(name=status)

I end up using this
BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = reader.readLine();
while (line != null) {
String[] value = line.split("=>");
if(value.length > 1){
if(value[0].contains("\"status\"")){
System.out.println(value[1]);
}
}
line = reader.readLine();
}
If anyone can suggest a better method would be greatly appreciated.

Is there a 'NOT IN' comparator in DynamoDB using query or scan

I read this document
http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Condition.html
maybe no condition equivalent not in of RDB just only exists in condition.
How can I implement not in or is there an equivalent to not in?

Here an implementation In PHP
public function getRecordsByInExpression($tableName, $providerCode, $fieldName, $values, $logicalOp = self::IN_EXP)
{
$queryExp =[
'TableName' => $tableName,
'KeyConditionExpression' => 'ProviderCode = :pc',
'FilterExpression' => $logicalOp == self::IN_EXP ? "$fieldName IN (:list)" : "NOT ($fieldName IN (:list))",
'ExpressionAttributeValues' => [
':pc' => $providerCode,
':list' => implode(',', $values)
]
];
return $this->query($queryExp)->get('Items');
}
At the end a NOT IN = NOT (field IN (:list))

Log analysis to query logs based on a log message

I have a Java application that ouputs log in the format
timestamp UUID1 some information
timestamp UUID1 some more information
timestamp UUID1 x = 1
timestamp UUID2 some information
timestamp UUID2 some more information
timestamp UUID2 x = 2
timestamp UUID3 some information
timestamp UUID3 some more information
timestamp UUID3 x = 1
I want to implement a log analysis framework using Elsatic Search, LogStash and Kibana. Is it possible to get the logs only according to X value?
For example:-
If I query X = 1, I should get only the following logs.
timestamp UUID1 some information
timestamp UUID1 some more information
timestamp UUID1 x = 1
timestamp UUID3 some information
timestamp UUID3 some more information
timestamp UUID3 x = 1
If I query X = 2, I should get only the following logs.
timestamp UUID2 some information
timestamp UUID2 some more information
timestamp UUID2 x = 2
I am in control of the log message format. If it is not directly popssible to do this query, I can change the message format also.
UPDATE 1:
I will be a little more specific.
The following are my log statements.
MDC.put("uuid", UUID.randomUUID().toString());
logger.info("Assigning value to the variable : {}", name);
this.setVal(value.getVal());
logger.info("{} = {}", name, value.getVal());
logger.info("Assigned value {} to the variable : {}", value.getVal(),
name);
MDC.clear();
I received the log statements in Logstash using UDP. And I am getting the messages like.
{
"#timestamp" => "2015-04-01T10:23:37.846+05:30",
"#version" => 1,
"message" => "Assigning value to the variable : X",
"logger_name" => "com.example.logstash.Variable",
"thread_name" => "pool-1-thread-1",
"level" => "INFO",
"level_value" => 20000,
"HOSTNAME" => "pnibinkj-W7-1",
"uuid" => "ab17b842-8348-4474-98e4-8bc2b8dd6781",
"host" => "127.0.0.1"
}
{
"#timestamp" => "2015-04-01T10:23:37.846+05:30",
"#version" => 1,
"message" => "Assigning value to the variable : Y",
"logger_name" => "com.example.logstash.Variable",
"thread_name" => "pool-1-thread-2",
"level" => "INFO",
"level_value" => 20000,
"HOSTNAME" => "pnibinkj-W7-1",
"uuid" => "d5513e4c-de3b-4144-87e4-87b077ac8056",
"host" => "127.0.0.1"
}
{
"#timestamp" => "2015-04-01T10:23:37.862+05:30",
"#version" => 1,
"message" => "Y = 1",
"logger_name" => "com.example.logstash.Variable",
"thread_name" => "pool-1-thread-2",
"level" => "INFO",
"level_value" => 20000,
"HOSTNAME" => "pnibinkj-W7-1",
"uuid" => "d5513e4c-de3b-4144-87e4-87b077ac8056",
"host" => "127.0.0.1"
}
{
"#timestamp" => "2015-04-01T10:23:37.863+05:30",
"#version" => 1,
"message" => "X = 1",
"logger_name" => "com.example.logstash.Variable",
"thread_name" => "pool-1-thread-1",
"level" => "INFO",
"level_value" => 20000,
"HOSTNAME" => "pnibinkj-W7-1",
"uuid" => "ab17b842-8348-4474-98e4-8bc2b8dd6781",
"host" => "127.0.0.1"
}
{
"#timestamp" => "2015-04-01T10:23:37.863+05:30",
"#version" => 1,
"message" => "Assigned value 1 to the variable : X",
"logger_name" => "com.example.logstash.Variable",
"thread_name" => "pool-1-thread-1",
"level" => "INFO",
"level_value" => 20000,
"HOSTNAME" => "pnibinkj-W7-1",
"uuid" => "ab17b842-8348-4474-98e4-8bc2b8dd6781",
"host" => "127.0.0.1"
}
{
"#timestamp" => "2015-04-01T10:23:37.863+05:30",
"#version" => 1,
"message" => "Assigned value 1 to the variable : Y",
"logger_name" => "com.example.logstash.Variable",
"thread_name" => "pool-1-thread-2",
"level" => "INFO",
"level_value" => 20000,
"HOSTNAME" => "pnibinkj-W7-1",
"uuid" => "d5513e4c-de3b-4144-87e4-87b077ac8056",
"host" => "127.0.0.1"
}
There are 2 UUIDs
"d5513e4c-de3b-4144-87e4-87b077ac8056" for "Y = 1"
"ab17b842-8348-4474-98e4-8bc2b8dd6781" for "X = 1"
There are two other messages for each UUID. I want to combine them into a single event.
I am not sure, how to write the multiline filter for this case.
filter {
multiline {
pattern => "."
what => "previous"
stream_identity => "%{uuid}"
}
}
"pattern" and "what" are required fields, it seems. What should I provide for these fields. How do I use Stream Identity?
Please point me in right direction.
Thanks,
Paul

You would need to combine your messages (see multiline{} filter, which supports stream_identity), and then a regular query would return the appropriate message.

this should be possible using the kibana filters if X is some unique value, but with the logs in the format shown you'd need to use the multiline filter to join the entries together.
With that in place, you could probably use a query something like
message: "X=1"

How to complete a Search in NetSuite using Java API?

I have a jar file nsws-2014_1.jar with the bindings for using Netsuite's web services. I am reading through their examples in the pdf which they have provide online, but the search or querying is confusing. I would like to complete a query that someone has done in perl, but in java. Can someone help me out?
my $query = {
basic => [
{ name => 'customFieldList', value => [
{
name => 'customField',
attr => {
scriptId => 'custentity_hexid',
operator => 'is',
'xsi:type' => 'core_2013_2.platform:SearchStringCustomField'
},
value => $json->{'ServiceOrder'}->{'Id'}
},
],
},
],
};

CustomerSearch customerSrch = new CustomerSearch();
CustomerSearchBasic customerSrchBsc = new CustomerSearchBasic();
SearchStringCustomField custentity_externalid = new SearchStringCustomField();
custentity_externalid.setInternalId("1015");
custentity_externalid.setSearchValue(externalID);
custentity_externalid.setOperator(SearchStringFieldOperator.is);
SearchCustomFieldList searchCustomFieldList = new SearchCustomFieldList();
SearchCustomField[] SearchCustomFieldArray = {custentity_externalid};
searchCustomFieldList.setCustomField(SearchCustomFieldArray);
customerSrchBsc.setCustomFieldList(searchCustomFieldList);
customerSrch.setBasic(customerSrchBsc);
SearchResult searchResult = _service.search(customerSrch);