I wanted to display XML data using logstash and Kibana in grid format. using below conf file I am able to display data into grid but not able to split row data.
Example:
Output
logstash.conf file :
input {
file {
path => "C:/ELK Stack/logstash-8.2.0-windows-x86_64/logstash-8.2.0/Test.xml"
start_position => "beginning"
sincedb_path => "NUL"
codec => multiline {
pattern => "^<?stations.*>"
negate => "true"
what => "previous"
auto_flush_interval => 1
max_lines => 3000
}}}
filter
{
xml
{
source => "message"
target => "parsed"
store_xml => "false"
xpath => [
"/stations/station/id/text()", "station_id",
"/stations/station/name/text()", "station_name"
]
}
mutate {
remove_field => [ "message"]
}
}
output {
elasticsearch {
action => "index"
hosts => "localhost:9200"
index => "logstash_index123xml"
workers => 1
}
stdout {
codec => rubydebug
}
}
xpath will always return arrays, to associate the members of the two arrays you are going to need to use a ruby filter. To get multiple events you can use a split filter to split an array which you build in the ruby filter. If you start with
<stations>
<station>
<id>1</id>
<name>a</name>
<id>2</id>
<name>b</name>
</station>
</stations>
then if you use
xml {
source => "message"
store_xml => "false"
xpath => {
"/stations/station/id/text()" => "[#metadata][station_id]"
"/stations/station/name/text()" => "[#metadata][station_name]"
}
remove_field => [ "message" ]
}
ruby {
code => '
ids = event.get("[#metadata][station_id]")
names = event.get("[#metadata][station_name]")
if ids.is_a? Array and names.is_a? Array y and ids.length == names.length
a = []
ids.each_index { |x|
a << { "station_name" => names[x], "station_id" => ids[x] }
}
event.set("[#metadata][theData]", a)
end
'
}
if [#metadata][theData] {
split {
field => "[#metadata][theData]"
add_field => {
"station_name" => "%{[#metadata][theData][station_name]}"
"station_id" => "%{[#metadata][theData][station_id]}"
}
}
}
You will get two events
{
"station_name" => "a",
"station_id" => "1",
...
}
{
"station_name" => "b",
"station_id" => "2",
...
}
I've been trying to validate if our server has started in Wildfly using the jboss-cli.bat
This is the command i'm using:
/host=slave-1/server-config=REST-server-one:read-resource(include-runtime=true)
and this is what i'm getting from the command
{
"outcome" => "success",
"result" => {
"auto-start" => true,
"cpu-affinity" => undefined,
"group" => "wildfly-server-group",
"name" => "wildfly-server",
"priority" => undefined,
"socket-binding-default-interface" => undefined,
"socket-binding-group" => undefined,
"socket-binding-port-offset" => 0,
"status" => "STARTED",
"update-auto-start-with-server-status" => false,
"interface" => undefined,
"jvm" => undefined,
"path" => undefined,
"ssl" => undefined,
"system-property" => undefined
}
Is there a command that will return the value of the status in that response?
You should be able to use the read-attribute operation.
/host=slave-1/server-config=REST-server-one:read-attribute(name=status)
I end up using this
BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = reader.readLine();
while (line != null) {
String[] value = line.split("=>");
if(value.length > 1){
if(value[0].contains("\"status\"")){
System.out.println(value[1]);
}
}
line = reader.readLine();
}
If anyone can suggest a better method would be greatly appreciated.
Im currently trying to receive this url in java (https://api.wynncraft.com/public_api.php?action=items&command=75)
The problem is, I can read any file ending in .json perfectly, but because of the .php (I think) it isnt working for this one.
Also, if someone could tell me how to get things like item_name into variables I can use? Would be great...
My code:
URL u;
try {
u = new URL("https://api.wynncraft.com/public_api.php?action=items&command=75");
URLConnection c = u.openConnection();
InputStream r = c.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(r));
for (String line; (line = reader.readLine()) != null;)
System.out.println(line);
} catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Download the json.api JAR from http://mvnrepository.com/artifact/org.json/json (or use Maven)
Change your code to read the entire string, not just a line.
Then do something like this to get at the values you want.
import org.json.*;
JSONParser JSON = new JSONParser();
JSONObject obj = JSON.parse(line);
String hr0 = obj.getJSONObject("0")
.getJSONObject("Identification")
.getString("health_regen");
This is a little cryptic, buy you should be able ascertain the variable values.
This is how the JSON would be structured if you were to use PHP:
$json = json_decode($data,true);
k1, k2 denote the level with in the structure.
-------------------------------
k0 request => array(
k1: timestamp => integer: $json['request']['timestamp'] => 1430494697
k1: ip => string: $json['request']['ip'] => 108.162.210.100
)
-------------------------------
k0 0 => array(
k1: item_name => string: $json[0]['item_name'] => Bob's Mythic Bow
k1: item_type => string: $json[0]['item_type'] => Legendary
k1: item_minecraft => string: $json[0]['item_minecraft'] => Bow
k1: item_min_lvl => string: $json[0]['item_min_lvl'] => 75
k1 identification => array(
k2: health_regen => string: $json[0]['identification']['health_regen'] => 1.5
k2: mana_regen => string: $json[0]['identification']['mana_regen'] => 1
k2: spell_dam => string: $json[0]['identification']['spell_dam'] => 30%
k2: life_steal => string: $json[0]['identification']['life_steal'] => 0.4
k2: mana_steal => string: $json[0]['identification']['mana_steal'] => 1
k2: xp_bonus => string: $json[0]['identification']['xp_bonus'] => 20%
k2: loot_bonus => string: $json[0]['identification']['loot_bonus'] => 20%
)
k1: min_dam => string: $json[0]['min_dam'] => 566
k1: max_dam => string: $json[0]['max_dam'] => 463
)
-------------------------------
k0 1 => array(
k1: item_name => string: $json[1]['item_name'] => Bob's Mythic Daggers
k1: item_type => string: $json[1]['item_type'] => Legendary
k1: item_minecraft => string: $json[1]['item_minecraft'] => Shears
k1: item_min_lvl => string: $json[1]['item_min_lvl'] => 75
k1 identification => array(
k2: health_regen => string: $json[1]['identification']['health_regen'] => 1.5
k2: mana_regen => string: $json[1]['identification']['mana_regen'] => 1
k2: spell_dam => string: $json[1]['identification']['spell_dam'] => 30%
k2: life_steal => string: $json[1]['identification']['life_steal'] => 0.4
k2: mana_steal => string: $json[1]['identification']['mana_steal'] => 1
k2: xp_bonus => string: $json[1]['identification']['xp_bonus'] => 20%
k2: loot_bonus => string: $json[1]['identification']['loot_bonus'] => 20%
)
k1: min_dam => string: $json[1]['min_dam'] => 420
k1: max_dam => string: $json[1]['max_dam'] => 389
)
-------------------------------
k0 2 => array(
k1: item_name => string: $json[2]['item_name'] => Bob's Mythic Spear
k1: item_type => string: $json[2]['item_type'] => Legendary
k1: item_minecraft => string: $json[2]['item_minecraft'] => Shovel
k1: item_min_lvl => string: $json[2]['item_min_lvl'] => 75
k1 identification => array(
k2: health_regen => string: $json[2]['identification']['health_regen'] => 1.5
k2: mana_regen => string: $json[2]['identification']['mana_regen'] => 1
k2: spell_dam => string: $json[2]['identification']['spell_dam'] => 30%
k2: life_steal => string: $json[2]['identification']['life_steal'] => 0.4
k2: mana_steal => string: $json[2]['identification']['mana_steal'] => 1
k2: xp_bonus => string: $json[2]['identification']['xp_bonus'] => 20%
k2: loot_bonus => string: $json[2]['identification']['loot_bonus'] => 20%
)
k1: min_dam => string: $json[2]['min_dam'] => 368
k1: max_dam => string: $json[2]['max_dam'] => 311
)
-------------------------------
k0 3 => array(
k1: item_name => string: $json[3]['item_name'] => Bob's Mythic Wand
k1: item_type => string: $json[3]['item_type'] => Legendary
k1: item_minecraft => string: $json[3]['item_minecraft'] => Stick
k1: item_min_lvl => string: $json[3]['item_min_lvl'] => 75
k1 identification => array(
k2: health_regen => string: $json[3]['identification']['health_regen'] => 1.5
k2: mana_regen => string: $json[3]['identification']['mana_regen'] => 1
k2: spell_dam => string: $json[3]['identification']['spell_dam'] => 30%
k2: life_steal => string: $json[3]['identification']['life_steal'] => 0.4
k2: mana_steal => string: $json[3]['identification']['mana_steal'] => 1
k2: xp_bonus => string: $json[3]['identification']['xp_bonus'] => 20%
k2: loot_bonus => string: $json[3]['identification']['loot_bonus'] => 20%
)
k1: min_dam => string: $json[3]['min_dam'] => 278
k1: max_dam => string: $json[3]['max_dam'] => 204
)
#################################################################
Summary of Array Constructs With Output Type
#################################################################
1 $json[int]['identification']['health_regen'] string
1 $json[int]['identification']['mana_regen'] string
1 $json[int]['identification']['spell_dam'] string
1 $json[int]['identification']['life_steal'] string
1 $json[int]['identification']['mana_steal'] string
1 $json[int]['identification']['xp_bonus'] string
1 $json[int]['identification']['loot_bonus'] string
I have a Java application that ouputs log in the format
timestamp UUID1 some information
timestamp UUID1 some more information
timestamp UUID1 x = 1
timestamp UUID2 some information
timestamp UUID2 some more information
timestamp UUID2 x = 2
timestamp UUID3 some information
timestamp UUID3 some more information
timestamp UUID3 x = 1
I want to implement a log analysis framework using Elsatic Search, LogStash and Kibana. Is it possible to get the logs only according to X value?
For example:-
If I query X = 1, I should get only the following logs.
timestamp UUID1 some information
timestamp UUID1 some more information
timestamp UUID1 x = 1
timestamp UUID3 some information
timestamp UUID3 some more information
timestamp UUID3 x = 1
If I query X = 2, I should get only the following logs.
timestamp UUID2 some information
timestamp UUID2 some more information
timestamp UUID2 x = 2
I am in control of the log message format. If it is not directly popssible to do this query, I can change the message format also.
UPDATE 1:
I will be a little more specific.
The following are my log statements.
MDC.put("uuid", UUID.randomUUID().toString());
logger.info("Assigning value to the variable : {}", name);
this.setVal(value.getVal());
logger.info("{} = {}", name, value.getVal());
logger.info("Assigned value {} to the variable : {}", value.getVal(),
name);
MDC.clear();
I received the log statements in Logstash using UDP. And I am getting the messages like.
{
"#timestamp" => "2015-04-01T10:23:37.846+05:30",
"#version" => 1,
"message" => "Assigning value to the variable : X",
"logger_name" => "com.example.logstash.Variable",
"thread_name" => "pool-1-thread-1",
"level" => "INFO",
"level_value" => 20000,
"HOSTNAME" => "pnibinkj-W7-1",
"uuid" => "ab17b842-8348-4474-98e4-8bc2b8dd6781",
"host" => "127.0.0.1"
}
{
"#timestamp" => "2015-04-01T10:23:37.846+05:30",
"#version" => 1,
"message" => "Assigning value to the variable : Y",
"logger_name" => "com.example.logstash.Variable",
"thread_name" => "pool-1-thread-2",
"level" => "INFO",
"level_value" => 20000,
"HOSTNAME" => "pnibinkj-W7-1",
"uuid" => "d5513e4c-de3b-4144-87e4-87b077ac8056",
"host" => "127.0.0.1"
}
{
"#timestamp" => "2015-04-01T10:23:37.862+05:30",
"#version" => 1,
"message" => "Y = 1",
"logger_name" => "com.example.logstash.Variable",
"thread_name" => "pool-1-thread-2",
"level" => "INFO",
"level_value" => 20000,
"HOSTNAME" => "pnibinkj-W7-1",
"uuid" => "d5513e4c-de3b-4144-87e4-87b077ac8056",
"host" => "127.0.0.1"
}
{
"#timestamp" => "2015-04-01T10:23:37.863+05:30",
"#version" => 1,
"message" => "X = 1",
"logger_name" => "com.example.logstash.Variable",
"thread_name" => "pool-1-thread-1",
"level" => "INFO",
"level_value" => 20000,
"HOSTNAME" => "pnibinkj-W7-1",
"uuid" => "ab17b842-8348-4474-98e4-8bc2b8dd6781",
"host" => "127.0.0.1"
}
{
"#timestamp" => "2015-04-01T10:23:37.863+05:30",
"#version" => 1,
"message" => "Assigned value 1 to the variable : X",
"logger_name" => "com.example.logstash.Variable",
"thread_name" => "pool-1-thread-1",
"level" => "INFO",
"level_value" => 20000,
"HOSTNAME" => "pnibinkj-W7-1",
"uuid" => "ab17b842-8348-4474-98e4-8bc2b8dd6781",
"host" => "127.0.0.1"
}
{
"#timestamp" => "2015-04-01T10:23:37.863+05:30",
"#version" => 1,
"message" => "Assigned value 1 to the variable : Y",
"logger_name" => "com.example.logstash.Variable",
"thread_name" => "pool-1-thread-2",
"level" => "INFO",
"level_value" => 20000,
"HOSTNAME" => "pnibinkj-W7-1",
"uuid" => "d5513e4c-de3b-4144-87e4-87b077ac8056",
"host" => "127.0.0.1"
}
There are 2 UUIDs
"d5513e4c-de3b-4144-87e4-87b077ac8056" for "Y = 1"
"ab17b842-8348-4474-98e4-8bc2b8dd6781" for "X = 1"
There are two other messages for each UUID. I want to combine them into a single event.
I am not sure, how to write the multiline filter for this case.
filter {
multiline {
pattern => "."
what => "previous"
stream_identity => "%{uuid}"
}
}
"pattern" and "what" are required fields, it seems. What should I provide for these fields. How do I use Stream Identity?
Please point me in right direction.
Thanks,
Paul
You would need to combine your messages (see multiline{} filter, which supports stream_identity), and then a regular query would return the appropriate message.
this should be possible using the kibana filters if X is some unique value, but with the logs in the format shown you'd need to use the multiline filter to join the entries together.
With that in place, you could probably use a query something like
message: "X=1"
I am getting this JSON response form my web server. as i searched that this format is json decode format. how i will convert this into json encode format.
stdClass Object
(
[id] => 4ffc88e7-1413-fa9c-423c-53fc701b1044
[entry_list] => stdClass Object
(
[first_name] => stdClass Object
(
[name] => first_name
[value] => dharmendra
)
[last_name] => stdClass Object
(
[name] => last_name
[value] => singh
)
[primary_address_city] => stdClass Object
(
[name] => primary_address_city
[value] => gwalior
)
[primary_address_street] => stdClass Object
(
[name] => primary_address_street
[value] => chinchwad
)
[primary_address_state] => stdClass Object
(
[name] => primary_address_state
[value] => mp
)
[phone_mobile] => stdClass Object
(
[name] => phone_mobile
[value] => 55555555
)
[primary_address_country] => stdClass Object
(
[name] => primary_address_country
[value] => in
)
[primary_address_postalcode] => stdClass Object
(
[name] => primary_address_postalcode
[value] => 4444444
)
)
)
this is json response which came from the SugerCRM RESTapi
the solution is write print(json_encode(set_result)) in the place of
print(set_result)