I am looking forward to know some best approaches to manage properties files.
We have a set of devices (say N). Each of these devices has certain properties.
e.g.
Device A has properties
A.a11=valuea11
A.a12=valuea12
.
Device B has properties
B.b11=valueb11
B.b12=valueb12
.
Apart from this they have some common properties applicable for all devices.
X.x11=valuex11
X.x12=valuex12
I am writing an automation for running some test suites on these devices. At a time, test script on run on a single device. The device name will be passed as a argument. Based on the device name, code will grab the respective properties and common properties and update the device with these properties. e.g. for device A, code will grab the A.a11, A.a12 (device A specific) and X.x11, X.x12 (common) properties and upload it to the device before running test script.
So, in code, I need to manage these properties so that only device specific and common properties will be uploaded to the device, ignoring the rest one. I am managing it like this
if ($device eq 'A') then
upload A's properties
elsif ($device eq 'B') then
upload B's properties
endif
upload Common (X) properties.
Managing device in this way is becoming little bit difficult as the number of devices are keep on increasing.
So I am looking forward for some other best approach to manage these properties.
This is a good case where v (aka traits in generalized OOP literature) will be useful.
Instead of the classical object is a class, with roles an object *does a * role.
Check out the appropriate Moose docs for way more information.
Example:
package Device::ActLikeA;
use Moose::Role;
has 'attribute' => (
isa => string,
is => 'rw',
default => 'Apple',
);
sub an_a_like_method {
my $self = shift;
# foo
}
1;
So now I have a role called Device::ActLikeA, what do I do with it?
Well, I can apply the role to a class, and the code and attributes defined in ActLikeA will be available in the class:
package Device::USBButterChurn;
use Moose;
does 'Device::ActLikeA';
# now has an attribute 'attribute' and a method 'an_a_like_method'
1;
You can also apply roles to individual instances of a class.
package Device;
use Moose;
has 'part_no' => (
isa => 'Str',
is => 'ro',
required => 1,
);
has 'serial' => {
isa => 'Str',
is => 'ro',
lazy => 1,
build => '_build_serial',
);
1;
And then main code that looks at the part and applies appropriate roles:
my #PART_MATCH = (
[ qr/Foo/, 'Device::MetaSyntacticVariable' ],
[ qr/^...-[^_]*[A][^-], 'Device::ActLikeA; ],
[ qr/^...-[^_]*[B][^-], 'Device::ActLikeB; ],
# etc
);
my $parts = load_parts($config_file);
for my $part ( #$parts ) {
my $part_no = $part->part_number();
for my $_ (#PART_MATCH) {
my ($match, $role) = #$_;
$part->apply_role($role)
if $part_no =~ /$match/;
}
}
Here is a very direct approach.
First of all you need a way to indicate that A "is-a" X and B "is-a" X, i.e. X is a parent of both A and B.
Then your upload_device routine will look something like this:
sub upload_properties {
my $device = shift;
... upload the "specific" properties of $device ...
for my $parent (parent's of $device) {
upload_properties($parent);
}
}
One implementation:
Indicate the "is-a" relationship with a line in your config file like:
A.isa = X
(Feel free to use some other syntax - what you use will depend on how you want to parse the file.)
From the config file, create a hash of all devices that looks like this:
$all_devices = {
A => { a11 => valuea11, a12 => valuea12, isa => [ 'X' ]},
B => { b11 => valueb11, b12 => valueb12, isa => [ 'X' ] },
X => { x11 => valuex11, x12 => valuex12, isa => [] },
}
The upload_properties routine:
sub upload_properties {
my ($device) = #_;
for my $key (keys %$device) {
next if $key eq "isa";
... upload property $key => $device->{$key} ...
}
my $isa = $device->{isa}; # this should be an array ref
for my $parent_name (#$isa) {
my $parent = $all_devices->{$parent_name};
upload_properties($parent);
}
}
# e.g. to upload device 'A':
upload_properties( $all_devices->{'A'} );
You can eliminate the large if-else chain by storing the device properties in a hash.
Then you need only ensure that the particular $device appears in that hash.
#!/usr/bin/perl
use warnings;
use strict;
my %vals = (
A => {
a11 => 'valuea11',
a12 => 'valuea12',
},
B => {
b11 => 'valueb11',
b12 => 'valueb12',
},
);
foreach my $device qw(A B C) {
if (exists $vals{$device}) {
upload_properties($vals{$device});
}
else {
warn "'$device' is not a valid device\n";
}
}
sub upload_properties {
my($h) = #_;
print "setting $_=$h->{$_}\n" for sort keys %$h; # simulate upload
print "\n";
}
Related
I wanted to display XML data using logstash and Kibana in grid format. using below conf file I am able to display data into grid but not able to split row data.
Example:
Output
logstash.conf file :
input {
file {
path => "C:/ELK Stack/logstash-8.2.0-windows-x86_64/logstash-8.2.0/Test.xml"
start_position => "beginning"
sincedb_path => "NUL"
codec => multiline {
pattern => "^<?stations.*>"
negate => "true"
what => "previous"
auto_flush_interval => 1
max_lines => 3000
}}}
filter
{
xml
{
source => "message"
target => "parsed"
store_xml => "false"
xpath => [
"/stations/station/id/text()", "station_id",
"/stations/station/name/text()", "station_name"
]
}
mutate {
remove_field => [ "message"]
}
}
output {
elasticsearch {
action => "index"
hosts => "localhost:9200"
index => "logstash_index123xml"
workers => 1
}
stdout {
codec => rubydebug
}
}
xpath will always return arrays, to associate the members of the two arrays you are going to need to use a ruby filter. To get multiple events you can use a split filter to split an array which you build in the ruby filter. If you start with
<stations>
<station>
<id>1</id>
<name>a</name>
<id>2</id>
<name>b</name>
</station>
</stations>
then if you use
xml {
source => "message"
store_xml => "false"
xpath => {
"/stations/station/id/text()" => "[#metadata][station_id]"
"/stations/station/name/text()" => "[#metadata][station_name]"
}
remove_field => [ "message" ]
}
ruby {
code => '
ids = event.get("[#metadata][station_id]")
names = event.get("[#metadata][station_name]")
if ids.is_a? Array and names.is_a? Array y and ids.length == names.length
a = []
ids.each_index { |x|
a << { "station_name" => names[x], "station_id" => ids[x] }
}
event.set("[#metadata][theData]", a)
end
'
}
if [#metadata][theData] {
split {
field => "[#metadata][theData]"
add_field => {
"station_name" => "%{[#metadata][theData][station_name]}"
"station_id" => "%{[#metadata][theData][station_id]}"
}
}
}
You will get two events
{
"station_name" => "a",
"station_id" => "1",
...
}
{
"station_name" => "b",
"station_id" => "2",
...
}
I have a java app running on appengine.
I'm logging my logs in json structure and then I can see my logs on stack driver (as in the docs)
package com.foo.bar;
public class MyClass {
private static final Logger log = Logger.getLogger(MyClass.class.getName());
public void myFunc() {
log.info("{msg: 'hello', corId: '123'}");
}
here is the message I get on stackdriver-logging:
com.foo.bar.MyClass myFunc: {msg: 'hello', corId: '123'}
and in the log-request object:
protoPayload.line[].logMessage = "com.foo.bar.MyClass myFunc: {msg: 'hello', corId: '123'}"
how can I make the log message to be only the message I am logging - without the class prefix:
{msg: 'hello', corId: '123'}
protoPayload.line[].logMessage = "{msg: 'hello', corId: '123'}"
I ended up shipping the logs from stackdriver to alasticsearch via logstash
in logstash I parsed my logs, I also seperated my logs tol have each log as its own record and not a nested array
see:
How to ship logs from pods on Kubernetes running on top of GCP to elasticsearch/logstash?
my config in logstash for parsing the logs:
filter {
if [resource][type] == "gae_app" {
# split the protoPayload.line array, so each log message is a separate entry in Elasticsearch
split {
field => "[protoPayload][line]"
target => "line"
remove_field => [ "httpRequest", "operation", "protoPayload"]
}
# extract `line.logMessage` and `line.severity` fields
mutate {
add_field => {"logMessage" => "%{[line][logMessage]}"}
replace => {"severity" => "%{[line][severity]}"}
remove_field => ["line"]
}
# remove the `com.example.MyClass myFunc: ` prefix from log
grok {
match => { "logMessage" => "^%{DATA}: %{GREEDYDATA:parsedMessage}"}
}
# parse the log message into json, json fields will be located in root
json {
source => "parsedMessage"
target => "jsonPayload"
add_field => {"[jsonPayload][level]" => "%{severity}"}
remove_field => ["parsedMessage", "logMessage"]
}
# uniform GAE logs to the structure of GKE logs
grok {
match => { # check..
"[resource][labels][version_id]" =>
"^%{DATA:[resource][labels][container_name]}-%{GREEDYDATA:[resource][labels][namespace_id]}"}
}
}
}
I read this document
http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Condition.html
maybe no condition equivalent not in of RDB just only exists in condition.
How can I implement not in or is there an equivalent to not in?
Here an implementation In PHP
public function getRecordsByInExpression($tableName, $providerCode, $fieldName, $values, $logicalOp = self::IN_EXP)
{
$queryExp =[
'TableName' => $tableName,
'KeyConditionExpression' => 'ProviderCode = :pc',
'FilterExpression' => $logicalOp == self::IN_EXP ? "$fieldName IN (:list)" : "NOT ($fieldName IN (:list))",
'ExpressionAttributeValues' => [
':pc' => $providerCode,
':list' => implode(',', $values)
]
];
return $this->query($queryExp)->get('Items');
}
At the end a NOT IN = NOT (field IN (:list))
I have a jar file nsws-2014_1.jar with the bindings for using Netsuite's web services. I am reading through their examples in the pdf which they have provide online, but the search or querying is confusing. I would like to complete a query that someone has done in perl, but in java. Can someone help me out?
my $query = {
basic => [
{ name => 'customFieldList', value => [
{
name => 'customField',
attr => {
scriptId => 'custentity_hexid',
operator => 'is',
'xsi:type' => 'core_2013_2.platform:SearchStringCustomField'
},
value => $json->{'ServiceOrder'}->{'Id'}
},
],
},
],
};
CustomerSearch customerSrch = new CustomerSearch();
CustomerSearchBasic customerSrchBsc = new CustomerSearchBasic();
SearchStringCustomField custentity_externalid = new SearchStringCustomField();
custentity_externalid.setInternalId("1015");
custentity_externalid.setSearchValue(externalID);
custentity_externalid.setOperator(SearchStringFieldOperator.is);
SearchCustomFieldList searchCustomFieldList = new SearchCustomFieldList();
SearchCustomField[] SearchCustomFieldArray = {custentity_externalid};
searchCustomFieldList.setCustomField(SearchCustomFieldArray);
customerSrchBsc.setCustomFieldList(searchCustomFieldList);
customerSrch.setBasic(customerSrchBsc);
SearchResult searchResult = _service.search(customerSrch);
Xampp comes with a neat executable called xampp-portcheck.exe. This responds with if the ports required are free, and if not, which applications are running on those ports.
I can check if something is running on a port, by accessing the netstat details, but how do I find out the application running on the port within Windows?
The CPAN module Win32::IPHelper provides access to GetExtendedTcpTable which provides the ProcessID for each connection.
Win32::Process::Info gives information about all running processes.
Combining the two, we get:
#!/usr/bin/perl
use strict;
use warnings;
use Win32;
use Win32::API;
use Win32::IPHelper;
use Win32::Process::Info qw( NT );
use Data::Dumper;
my #tcptable;
Win32::IPHelper::GetExtendedTcpTable(\#tcptable, 1);
my $pi = Win32::Process::Info->new;
my %pinfo = map {$_->{ProcessId} => $_ } $pi->GetProcInfo;
for my $conn ( #tcptable ) {
my $pid = $conn->{ProcessId};
$conn->{ProcessName} = $pinfo{$pid}->{Name};
$conn->{ProcessExecutablePath} = $pinfo{$pid}->{ExecutablePath};
}
#tcptable =
sort { $a->[0] cmp $b->[0] }
map {[ sprintf("%s:%s", $_->{LocalAddr}, $_->{LocalPort}) => $_ ]}
#tcptable;
print Dumper \#tcptable;
Output:
[
'0.0.0.0:135',
{
'RemotePort' => 0,
'LocalPort' => 135,
'LocalAddr' => '0.0.0.0',
'State' => 'LISTENING',
'ProcessId' => 1836,
'ProcessName' => 'svchost.exe',
'ProcessExecutablePath' => 'C:\\WINDOWS\\system32\\svchost.exe',
'RemoteAddr' => '0.0.0.0'
}
],
...
[
'192.168.169.150:1841',
{
'RemotePort' => 80,
'LocalPort' => 1841,
'LocalAddr' => '192.168.169.150',
'State' => 'ESTABLISHED',
'ProcessId' => 1868,
'ProcessName' => 'firefox.exe',
'ProcessExecutablePath' => 'C:\\Program Files\\Mozilla Firefox\\firefox.exe',
'RemoteAddr' => '69.59.196.211'
}
],
Phewwww it was exhausting connecting all these dots.