I am working on a Java project to parse logs where I am using Java Grok library for pattern recognition. I have given the pattern as follows:
%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}
When I try parsing the line,
Dec 23 14:30:01 louis CRON[619]: (www-data) CMD (php /usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log)
it gives the following output:
syslog_timestamp=Dec 23 14:30:01
syslog_hostname=louis
syslog_program=CRON
syslog_pid=619
syslog_message=(www-data) CMD (php /usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log)
I want to extract details from the syslog_message further to get stuff like facility and severity. How can I improve the grok pattern to get these details?
I am not an expert of Java Grok. You can use www-data as your facility and error as your severity.
Change your pattern to get this information.
syslog_facility=www-data
syslog-severity=error
Regards,
Ivo
As far as I see, you can't get the severity and facility from that message because it's not there. Assuming this is written by a syslog daemon, and assuming that syslog daemon is rsyslog, you can change the way things are written down by altering the template and select the properties you want.
You can write, for example, one JSON per line in a file with a template like this (basically copy-pasted from this rsyslog->Elasticsearch blog post):
template(name="jsonPerLine"
type="list") {
constant(value="{")
constant(value="\"#timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\"}\n")
}
This way, in your application you can parse this JSON and get all these fields, which is much cheaper than using regular expressions (which is what grok does).
You'd use this template to write to files later on in the configuration, for example:
action(
type="omfile"
file="/var/log/json_messages"
template="jsonPerLine"
)
If you get errors from the syntax above, it may be that your distro has a really-really old version of rsyslog. If that's so, you can get packages from the official repositories.
Related
I am using Netlogo Api Controller With spring boot
this my code (i got it from this link )
HeadlessWorkspace workspace = HeadlessWorkspace.newInstance();
try {
workspace.open("models/Residential_Solar_PV_Adoption.nlogo",true);
workspace.command("set number-of-residences 900");
workspace.command("set %-similar-wanted 7");
workspace.command("set count-years-simulated 14");
workspace.command("set number-of-residences 500");
workspace.command("set carbon-tax 13.7");
workspace.command("setup");
workspace.command("repeat 10 [ go ]");
workspace.command("reset-ticks");
workspace.dispose();
workspace.dispose();
}
catch(Exception ex) {
ex.printStackTrace();
}
i got this result in the console:
But I want to get the table view and save to database. Which command can I use to get the table view ?
Table view:
any help please ?
If you can clarify why you're trying to generate the data this way, I or others might be able to give better advice.
There is no single NetLogo command or NetLogo API method to generate that table, you have to use BehaviorSpace to get it. Here are some options, listed in rough order of simplest to hardest.
Option 1
If possible, I'd recommend just running BehaviorSpace experiments from the command line to generate your table. This will get you exactly the same output you're looking for. You can find information on how to do that in the NetLogo manual's BehaviorSpace guide. If necessary, you can run NetLogo headless from the command line from within a Java program, just look for resources on calling out to external programs from Java, maybe with ProcessBuilder.
If you're running from within Java in order to setup and change the parameters of your BehaviorSpace experiments in a way that you cannot do from within the program, you could instead generate experiment XML files in Java to pass to NetLogo at the command line. See the docs on the XML format.
Option 2
You can recreate the contents of the table using the CSV extension in your model and adding a few more commands to generate the data. This will not create the exact same table, but it will get your data output in a computer and human readable format.
In pure NetLogo code, you'd want something like the below. Note that you can control more of the behavior (like file names or the desired variables) by running other pre-experiment commands before running setup or go in your Java code. You could also run the CSV-specific file code from Java using the controlling API and leave the model unchanged, but you'll need to write your own NetLogo code version of the csv:to-row primitive.
globals [
;; your model globals here
output-variables
]
to setup
clear-all
;;; your model setup code here
file-open "my-output.csv"
; the given variables should be valid reporters for the NetLogo model
set output-variables [ "ticks" "current-price" "number-of-residences" "count-years-simulated" "solar-PV-cost" "%-lows" "k" ]
file-print csv:to-row output-variables
reset-ticks
end
to go
;;; the rest of your model code here
file-print csv:to-row map [ v -> runresult v ] output-variables
file-flush
tick
end
Option 3
If you really need to reproduce the BehaviorSpace table export exactly, you can try to run a BehaviorSpace experiment directly from Java. The table is generated by this code but as you can see it's tied in with the LabProtocol class, meaning you'll have to setup and run your model through BehaviorSpace instead of just step-by-step using a workspace as you've done in your sample code.
A good example of this might be the Main.scala object, which extracts some experiment settings from the expected command-line arguments, and then uses them with the lab.run() method to run the BehaviorSpace experiment and generate the output. That's Scala code and not Java, but hopefully it isn't too hard to translate. You'd similarly have to setup an org.nlogo.nvm.LabInterface.Settings instance and pass that off to a HeadlessWorkspace.newLab.run() to get things going.
My requirement is simple. At the beginning of each file there should be a block comment like this:
/*
* This file was last modified by {username} at {date} and has revision number {revisionnumber}
*/
I want to populate the {username}, {date} and {revisionnumber} with the appropriate content from SVN.
How can I achieve this with NetBeans and Subversion? I have searched a lot but I can't find exactly what I need.
I looked at this question and got some useful information. It is not exactly duplicate because I am working with NetBeans but the idea is the same. This is my header:
/*
* $LastChangedDate$
* $LastChangedRevision$
*/
Then I go to Team > Subversion > Svn properties and add svn:keywords as property name and LastChangedDate LastChangedRevision as property value.
And when I commit from NetBeans it looks like this:
/*
* $LastChangedDate: 2012-02-13 17:38:57 +0200 (Пн, 13 II 2012) $
* $LastChangedRevision: 27 $
*/
Thanks all for the support! I will accept my answer because other answers do not include the NetBeans information. Nevertheless I give +1 to the other answers.
As this data only exists after the file was committed it should be set by SVN itself, not a client program. (And client-side processing tends to get disabled or not configured at all.) This means there is no simple template/substitute like you want, because then after the first replacement the template variables would be lost.
You can find information abut SVN's keyword substitution here. Then things like $Rev$ can be replaced by $Rev: 12 $.
You can do this with The SubWCRev Program.
SubWCRev is Windows console program which can be used to read the
status of a Subversion working copy and optionally perform keyword
substitution in a template file. This is often used as part of the
build process as a means of incorporating working copy information
into the object you are building. Typically it might be used to
include the revision number in an “About” box.
This is typically done during the build process.
If you use Linux, you can find a Linux binary here. If you wish, you could also write your own using the output of svn log.
I followed Petar Minchev's suggestions, only I put the $LastChangedRevision$ tag not in a comment block but embedded it in a string. Now it is available to programmatically display the revision number in a Help -> About dialog.
String build = "$LastChangedRevision$";
I can later display the revision value in the about dialog using a String that has all of the fluff trimmed off.
String version = build.replace("$LastChangedRevision:", "").replace("$", "").trim();
I recommend a slightly different approach.
Put the following header at the top of your source files.
/*
* This file was last modified by {username} at {date} and has revision number {revisionnumber}
*/
Then add a shell script like this
post update, checkout script
USERNAME=# // use svnversion to get username
DATE=# // use svnversion to get revisio nnumber
sed -e "s#{username}#${USERNAME}#" -e "s#{date}#${DATE}#" ${SOURCE_CONTROL_FILE} > ${SOURCE_FILE}
pre commit script
cat standard_header.txt > ${SOURCE_CONTROL_FILE}
tail --lines $((${LENGTH}-4)) ${SOURCE_FILE} >> ${SOURCE_CONTROL_FILE}
I am writing a trading program in perl with the newest Finance::InteractiveBrokers::TWS
module. I kick off a command line interface in a separate thread at the
beginning of my program but then when I try to create a tws object, my program
exits with this message:
As of Inline v0.30, use of the Inline::Config module is no longer supported or
allowed. If Inline::Config exists on your system, it can be removed. See the
Inline documentation for information on how to configure Inline. (You should
find it much more straightforward than Inline::Config :-)
I have the newest versions of Inline and Inline::Java. I looked at TWS.pm and it doesn't seem to be using Inline::Config. I set 'SHARED_JVM => 1' in the 'use Inline()' and 'Inline->bind()' calls in TWS.pm but that did not resolve the issue...
My Code:
use Finance::InteractiveBrokers::TWS;
use threads;
use threads::shared;
our $callback;
our $tws;
my $interface = UserInterface->new();
share($interface);
my $t = threads->create(sub{$interface->runUI()});
$callback= TWScallback->new();
$tws = Finance::InteractiveBrokers::TWS->new($manager); #This is where the program fails
So is Inline::Config installed on your system or not? A cursory inspection of the code is not sufficient to tell whether Perl is loading a module or not. There are too many esoteric ways (some intentional and some otherwise) to load a package or otherwise populate a namespace.
The error message in question comes from this line of code in Inline.pm:
croak M14_usage_Config() if %main::Inline::Config::;
so something in your program is populating the Inline::Config namespace. You should do what the program instructs you to do: find out where Inline/Config.pm is installed on your system (somewhere in your #INC path) and delete it.
well... i have a file containing tintin-script. Now i already managed to grab all actions and substitutions from it to show them properly ordered on a website using Ruby, which helps me to keep an overview.
Example TINTIN-script
#substitution {You tell {([a-zA-Z,\-\ ]*)}, %*$}
{<279>[<269> $sysdate[1]<279>, <269>$systime<279> |<219> Tell <279>] <269>to <219>%2<279> : <219>%3}
{4}
#substitution {{([a-zA-Z,\-\ ]*)} tells you, %*$}
{<279>[<269> $sysdate[1]<279>, <269>$systime<279> |<119> Tell <279>] <269>from <119>%2<279> : <119>%3}
{2}
#action {Your muscles suddenly relax, and your nimbleness is gone.}
{
#if {$sw_keepaon}
{
aon;
};
} {5}
#action {xxxxx}
{
#if {$sw_keepfamiliar}
{
familiar $familiar;
};
} {5}
To grab them in my Ruby-App i read my script-file into a varibable 'input' and then use the following pattern to scan the 'input'
pattern = /(?<braces>{([^{}]|\g<braces>)*}){0}^#(?<type>action|substitution)\s*(?<b1>\g<braces>)\s*(?<b2>\g<braces>)\s*(?<b3>\g<braces>)/im
input = ""
File.open("/home/igambin/lmud/lmud.tt") { |file| input = file.read }
input.scan(pattern) { |prio, type, pattern, code|
## here i usually create objects, but for simplicity only output now
puts "Type : #{type}"
puts "Pattern : #{pattern}"
puts "Priority: #{prio}"
puts "Code :\n#{code}"
puts
}
Now my idea was to use the netbeans platform to write a module to not only keep an overview but also to assist editing the tintin script file. So opening the file in an Editor-Window I still need to parse the tintin-file and have all 'actions' and 'substitutions' from the file grabbed and displayed in an eTable, in wich I could dbl-click on one item to open a modification-window.
I've setup the module and got everything ready so far, i just can't figure out how to translate the ruby-regex pattern i've written to a working java-regex-pattern. It seems named-group-capturing and especially the recursive application of these groups is not supported in Java. Without that I seem to be unable to find a working solution...
Here's the ruby pattern again...
pattern = /(?<braces>{([^{}]|\g<braces>)*}){0}^#(?<type>action|substitution)\s*(?<b1>\g<braces>)\s*(?<b2>\g<braces>)\s*(?<b3>\g<braces>)/im
Can anyone help me to create a java pattern that matches the same?
Many thanks in advance for tips/hints/ideas and especially for solutions or (close-to-solution comments)!
Your text format seems pretty simple; it's possible you don't really need recursive matching. This Java-compatible regex matches your sample data correctly, as far as I can tell:
(?s)#(substitution|action)\s*\{(.*?)\}\s*\{(.*?)\}\s*\{(\d+)\}
Would that work for you? If you run Java 7, you can even name the groups. ;)
Can anyone help me to create a java pattern that matches the same?
No, no one can: Java's regex engine does not support recursive patterns (as Ruby 1.9 does).
I'm trying to use boilerpipe from JRuby. I've seen the guide for calling Java from JRuby, and have used it successfully with another Java package, but can't figure out why the same thing isn't working with boilerpipe.
I'm trying to basically do the equivalent of this Java from JRuby:
URL url = new URL("http://www.example.com/some-location/index.html");
String text = ArticleExtractor.INSTANCE.getText(url);
Tried this in JRuby:
require 'java'
url = java.net.URL.new("http://www.example.com/some-location/index.html")
text = Java::DeL3sBoilerpipeExtractors::ArticleExtractor.INSTANCE.getText(url)
This is based on the API Javadocs for boilerpipe. Here's the error:
jruby-1.6.0 :042 > Java::DeL3sBoilerpipeExtractors::ArticleExtractor
NameError: cannot load Java class deL3sBoilerpipeExtractors.ArticleExtractor
from org/jruby/javasupport/JavaClass.java:1195:in `for_name'
from org/jruby/javasupport/JavaUtilities.java:34:in `get_proxy_class'
from /usr/local/rvm/rubies/jruby-1.6.0/lib/ruby/site_ruby/shared/builtin/javasupport/java.rb:45:in `const_missing'
from (irb):42:in `evaluate'
from org/jruby/RubyKernel.java:1087:in `eval'
from /usr/local/rvm/rubies/jruby-1.6.0/lib/ruby/1.8/irb.rb:158:in `eval_input'
from /usr/local/rvm/rubies/jruby-1.6.0/lib/ruby/1.8/irb.rb:271:in `signal_status'
from /usr/local/rvm/rubies/jruby-1.6.0/lib/ruby/1.8/irb.rb:270:in `signal_status'
from /usr/local/rvm/rubies/jruby-1.6.0/lib/ruby/1.8/irb.rb:155:in `eval_input'
from org/jruby/RubyKernel.java:1417:in `loop'
from org/jruby/RubyKernel.java:1190:in `catch'
from /usr/local/rvm/rubies/jruby-1.6.0/lib/ruby/1.8/irb.rb:154:in `eval_input'
from /usr/local/rvm/rubies/jruby-1.6.0/lib/ruby/1.8/irb.rb:71:in `start'
from org/jruby/RubyKernel.java:1190:in `catch'
from /usr/local/rvm/rubies/jruby-1.6.0/lib/ruby/1.8/irb.rb:70:in `start'
from /usr/local/rvm/rubies/jruby-1.6.0/bin/irb:17:in `(root)'
Looks like it didn't parse the camelcase into the appropriate Java package name. What am I doing wrong? I believe I've set up my classpath alright (last 3 entries), though there may be some conflict with xerces possibly being included twice:
$ echo $CLASSPATH
:/jellly/Maui1.2:/jellly/Maui1.2/src:/jellly/Maui1.2/bin:/jellly/Maui1.2/lib/commons-io-1.4.jar:/jellly/Maui1.2/lib/commons-logging.jar:/jellly/Maui1.2/lib/icu4j_3_4.jar:/jellly/Maui1.2/lib/iri.jar:/jellly/Maui1.2/lib/jena.jar:/jellly/Maui1.2/lib/maxent-2.4.0.jar:/jellly/Maui1.2/lib/mysql-connector-java-3.1.13-bin.jar:/jellly/Maui1.2/lib/opennlp-tools-1.3.0.jar:/jellly/Maui1.2/lib/snowball.jar:/jellly/Maui1.2/lib/trove.jar:/jellly/Maui1.2/lib/weka.jar:/jellly/Maui1.2/lib/wikipediaminer1.1.jar:/jellly/Maui1.2/lib/xercesImpl.jar:/jellly/boilerpipe-1.1.0/boilerpipe-1.1.0.jar:/jellly/boilerpipe-1.1.0/lib/nekohtml-1.9.13.jar:/jellly/boilerpipe-1.1.0/lib/xerces-2.9.1.jar
I'd recommend against trying to guess the module name we put under Java::, since for unusual packages it can get mangled pretty badly. Use java_import 'your.weird.package.ArticleExtractor' or if all the package components are compatible with Ruby method naming, you can also do Java::your.weird.package.ArticleExtractor.
Also, since you might run into this... you'll want to reference the INSTANCE variable as ArticleExtractor::INSTANCE, since we map it as a Ruby constant.
Have fun!
You can also use the nice Jruby Boilerpipe Gem which wraps the Java code
Or the pure ruby implementation of Boilerpipe Ruby Boilerpipe Gem
require 'boilerpipe'
Boilerpipe::Extractors::ArticleExtractor.text("https://github.com/jruby/jruby/wiki/AboutJRuby")