We Keep Coding

XML data display in grid using kibana and logstash

I wanted to display XML data using logstash and Kibana in grid format. using below conf file I am able to display data into grid but not able to split row data.
Example:
Output
logstash.conf file :
input {
file {
path => "C:/ELK Stack/logstash-8.2.0-windows-x86_64/logstash-8.2.0/Test.xml"
start_position => "beginning"
sincedb_path => "NUL"
codec => multiline {
pattern => "^<?stations.*>"
negate => "true"
what => "previous"
auto_flush_interval => 1
max_lines => 3000
}}}
filter
{
xml
{
source => "message"
target => "parsed"
store_xml => "false"
xpath => [
"/stations/station/id/text()", "station_id",
"/stations/station/name/text()", "station_name"
]
}
mutate {
remove_field => [ "message"]
}
}
output {
elasticsearch {
action => "index"
hosts => "localhost:9200"
index => "logstash_index123xml"
workers => 1
}
stdout {
codec => rubydebug
}
}

xpath will always return arrays, to associate the members of the two arrays you are going to need to use a ruby filter. To get multiple events you can use a split filter to split an array which you build in the ruby filter. If you start with
<stations>
<station>
<id>1</id>
<name>a</name>
<id>2</id>
<name>b</name>
</station>
</stations>
then if you use
xml {
source => "message"
store_xml => "false"
xpath => {
"/stations/station/id/text()" => "[#metadata][station_id]"
"/stations/station/name/text()" => "[#metadata][station_name]"
}
remove_field => [ "message" ]
}
ruby {
code => '
ids = event.get("[#metadata][station_id]")
names = event.get("[#metadata][station_name]")
if ids.is_a? Array and names.is_a? Array y and ids.length == names.length
a = []
ids.each_index { |x|
a << { "station_name" => names[x], "station_id" => ids[x] }
}
event.set("[#metadata][theData]", a)
end
'
}
if [#metadata][theData] {
split {
field => "[#metadata][theData]"
add_field => {
"station_name" => "%{[#metadata][theData][station_name]}"
"station_id" => "%{[#metadata][theData][station_id]}"
}
}
}
You will get two events
{
"station_name" => "a",
"station_id" => "1",
...
}
{
"station_name" => "b",
"station_id" => "2",
...
}

Is there a 'NOT IN' comparator in DynamoDB using query or scan

I read this document
http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Condition.html
maybe no condition equivalent not in of RDB just only exists in condition.
How can I implement not in or is there an equivalent to not in?

Here an implementation In PHP
public function getRecordsByInExpression($tableName, $providerCode, $fieldName, $values, $logicalOp = self::IN_EXP)
{
$queryExp =[
'TableName' => $tableName,
'KeyConditionExpression' => 'ProviderCode = :pc',
'FilterExpression' => $logicalOp == self::IN_EXP ? "$fieldName IN (:list)" : "NOT ($fieldName IN (:list))",
'ExpressionAttributeValues' => [
':pc' => $providerCode,
':list' => implode(',', $values)
]
];
return $this->query($queryExp)->get('Items');
}
At the end a NOT IN = NOT (field IN (:list))

creating JSON format output [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I am trying to write a program which can create an output in JSON format, how would be best way of doing this? and programming languages?
This is an example output of JSON (expected output) which I need to input in the Name, Gender, Qualification and other attributes in a user friendly way during the execution of script. And in which outputs in following JSON format. Sorry, I am new in programming, but so much interested to learn Perl (or) Python (or) Java. What could be the best here?
Any suggestions?
P.S Sorry I am quite new to JSON as well, please apologize me for this basic one.
[
{
"Name":"Steven Mark",
"gender":"male",
"Qualification": {
"college":"Bachelor in Science",
"tech":"certified pro"
},
"contributions": [
{
"name":"biography",
"type":"book",
},
]
},
{
"Name":"Andrea Mark",
"Gender":"female",
"Qualifications": {
"college":"Bachelor in physics",
},
"contributions": [
{
"name":"my insights",
"type":"movie",
},
]
}
]

Virtually every language has a JSON library, including Perl.
use JSON;
my $data = [
{
"Name" => "Steven Mark",
"gender" => "male",
"Qualification" => {
"college" => "Bachelor in Science",
"tech" => "certified pro"
},
"contributions" => [
{
"name" => "biography",
"type" => "book",
},
]
},
{
"Name" => "Andrea Mark",
"Gender" => "female",
"Qualifications" => {
"college" => "Bachelor in physics",
},
"contributions" => [
{
"name" => "my insights",
"type" => "movie",
},
]
}
];
print(encode_json($data));

If you agree to use ANY programming language, i can suggest python. With its json lib you can do following (lines with # is comments):
# import lib
import json
# fill data into variable (this is list with dict objects inside):
data = [{"name":"john"},{"name": "bill"}]
# dump json
json.dumps(data)
Which will output your data as json.
You can start writing python using something from https://wiki.python.org/moin/BeginnersGuide

If you are going to use Python, you can try to use simplejson or json module to create a json object.
For example,
try:
import simplejson
except:
import json
data = dict(a=1,b=2)
with open("results.json", "w") as fp:
json.dump(data, fp, indent=3, encoding="utf-8")
For dumping, json is faster than simplejson (but not by an order of magnitude). For loading, simplejson is faster (but not by an order of magnitude).
You can check here for more detailed comparison between simplejson and json.

best practices to managing and loading properties

I am looking forward to know some best approaches to manage properties files.
We have a set of devices (say N). Each of these devices has certain properties.
e.g.
Device A has properties
A.a11=valuea11
A.a12=valuea12
.
Device B has properties
B.b11=valueb11
B.b12=valueb12
.
Apart from this they have some common properties applicable for all devices.
X.x11=valuex11
X.x12=valuex12
I am writing an automation for running some test suites on these devices. At a time, test script on run on a single device. The device name will be passed as a argument. Based on the device name, code will grab the respective properties and common properties and update the device with these properties. e.g. for device A, code will grab the A.a11, A.a12 (device A specific) and X.x11, X.x12 (common) properties and upload it to the device before running test script.
So, in code, I need to manage these properties so that only device specific and common properties will be uploaded to the device, ignoring the rest one. I am managing it like this
if ($device eq 'A') then
upload A's properties
elsif ($device eq 'B') then
upload B's properties
endif
upload Common (X) properties.
Managing device in this way is becoming little bit difficult as the number of devices are keep on increasing.
So I am looking forward for some other best approach to manage these properties.

This is a good case where v (aka traits in generalized OOP literature) will be useful.
Instead of the classical object is a class, with roles an object *does a * role.
Check out the appropriate Moose docs for way more information.
Example:
package Device::ActLikeA;
use Moose::Role;
has 'attribute' => (
isa => string,
is => 'rw',
default => 'Apple',
);
sub an_a_like_method {
my $self = shift;
# foo
}
1;
So now I have a role called Device::ActLikeA, what do I do with it?
Well, I can apply the role to a class, and the code and attributes defined in ActLikeA will be available in the class:
package Device::USBButterChurn;
use Moose;
does 'Device::ActLikeA';
# now has an attribute 'attribute' and a method 'an_a_like_method'
1;
You can also apply roles to individual instances of a class.
package Device;
use Moose;
has 'part_no' => (
isa => 'Str',
is => 'ro',
required => 1,
);
has 'serial' => {
isa => 'Str',
is => 'ro',
lazy => 1,
build => '_build_serial',
);
1;
And then main code that looks at the part and applies appropriate roles:
my #PART_MATCH = (
[ qr/Foo/, 'Device::MetaSyntacticVariable' ],
[ qr/^...-[^_]*[A][^-], 'Device::ActLikeA; ],
[ qr/^...-[^_]*[B][^-], 'Device::ActLikeB; ],
# etc
);
my $parts = load_parts($config_file);
for my $part ( #$parts ) {
my $part_no = $part->part_number();
for my $_ (#PART_MATCH) {
my ($match, $role) = #$_;
$part->apply_role($role)
if $part_no =~ /$match/;
}
}

Here is a very direct approach.
First of all you need a way to indicate that A "is-a" X and B "is-a" X, i.e. X is a parent of both A and B.
Then your upload_device routine will look something like this:
sub upload_properties {
my $device = shift;
... upload the "specific" properties of $device ...
for my $parent (parent's of $device) {
upload_properties($parent);
}
}
One implementation:
Indicate the "is-a" relationship with a line in your config file like:
A.isa = X
(Feel free to use some other syntax - what you use will depend on how you want to parse the file.)
From the config file, create a hash of all devices that looks like this:
$all_devices = {
A => { a11 => valuea11, a12 => valuea12, isa => [ 'X' ]},
B => { b11 => valueb11, b12 => valueb12, isa => [ 'X' ] },
X => { x11 => valuex11, x12 => valuex12, isa => [] },
}
The upload_properties routine:
sub upload_properties {
my ($device) = #_;
for my $key (keys %$device) {
next if $key eq "isa";
... upload property $key => $device->{$key} ...
}
my $isa = $device->{isa}; # this should be an array ref
for my $parent_name (#$isa) {
my $parent = $all_devices->{$parent_name};
upload_properties($parent);
}
}
# e.g. to upload device 'A':
upload_properties( $all_devices->{'A'} );

You can eliminate the large if-else chain by storing the device properties in a hash.
Then you need only ensure that the particular $device appears in that hash.
#!/usr/bin/perl
use warnings;
use strict;
my %vals = (
A => {
a11 => 'valuea11',
a12 => 'valuea12',
},
B => {
b11 => 'valueb11',
b12 => 'valueb12',
},
);
foreach my $device qw(A B C) {
if (exists $vals{$device}) {
upload_properties($vals{$device});
}
else {
warn "'$device' is not a valid device\n";
}
}
sub upload_properties {
my($h) = #_;
print "setting $_=$h->{$_}\n" for sort keys %$h; # simulate upload
print "\n";
}

Name of applications running on port in Perl or Java

Xampp comes with a neat executable called xampp-portcheck.exe. This responds with if the ports required are free, and if not, which applications are running on those ports.
I can check if something is running on a port, by accessing the netstat details, but how do I find out the application running on the port within Windows?

The CPAN module Win32::IPHelper provides access to GetExtendedTcpTable which provides the ProcessID for each connection.
Win32::Process::Info gives information about all running processes.
Combining the two, we get:
#!/usr/bin/perl
use strict;
use warnings;
use Win32;
use Win32::API;
use Win32::IPHelper;
use Win32::Process::Info qw( NT );
use Data::Dumper;
my #tcptable;
Win32::IPHelper::GetExtendedTcpTable(\#tcptable, 1);
my $pi = Win32::Process::Info->new;
my %pinfo = map {$_->{ProcessId} => $_ } $pi->GetProcInfo;
for my $conn ( #tcptable ) {
my $pid = $conn->{ProcessId};
$conn->{ProcessName} = $pinfo{$pid}->{Name};
$conn->{ProcessExecutablePath} = $pinfo{$pid}->{ExecutablePath};
}
#tcptable =
sort { $a->[0] cmp $b->[0] }
map {[ sprintf("%s:%s", $_->{LocalAddr}, $_->{LocalPort}) => $_ ]}
#tcptable;
print Dumper \#tcptable;
Output:
[
'0.0.0.0:135',
{
'RemotePort' => 0,
'LocalPort' => 135,
'LocalAddr' => '0.0.0.0',
'State' => 'LISTENING',
'ProcessId' => 1836,
'ProcessName' => 'svchost.exe',
'ProcessExecutablePath' => 'C:\\WINDOWS\\system32\\svchost.exe',
'RemoteAddr' => '0.0.0.0'
}
],
...
[
'192.168.169.150:1841',
{
'RemotePort' => 80,
'LocalPort' => 1841,
'LocalAddr' => '192.168.169.150',
'State' => 'ESTABLISHED',
'ProcessId' => 1868,
'ProcessName' => 'firefox.exe',
'ProcessExecutablePath' => 'C:\\Program Files\\Mozilla Firefox\\firefox.exe',
'RemoteAddr' => '69.59.196.211'
}
],
Phewwww it was exhausting connecting all these dots.