How to enable/setup log4j for oozi java workflows? - java

I'm running an Oozie Java workflow (the jar file is in HDFS), and I'd like to add logging functionality to my application. Does anybody know how to do it? Where should I put my "log4j.properties" file? How can I make log4j to output the log to a location in HDFS?

Looking in this documentation, you can try adding oozie-log4j.properties in your oozie directory (where workflow.xml is).
Here are the default settings:
log4j.appender.oozie=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.oozie.RollingPolicy=org.apache.oozie.util.OozieRollingPolicy
log4j.appender.oozie.File=${oozie.log.dir}/oozie.log
log4j.appender.oozie.Append=true
log4j.appender.oozie.layout=org.apache.log4j.PatternLayout
log4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n
log4j.appender.oozie.RollingPolicy.FileNamePattern=${log4j.appender.oozie.File}-%d{yyyy-MM-dd-HH}
log4j.appender.oozie.RollingPolicy.MaxHistory=720
It also outlines the following restrictions:
In order for Oozie logging to work 100% correctly, the following restrictions must be observed (described below and in the oozie-log4j.properties file):
The appender that Oozie uses must be named "oozie" (i.e. log4j.appender.oozie )
log4j.appender.oozie.RollingPolicy.FileNamePattern must end with "-%d{yyyy-MM-dd-HH}.gz" or "-%d{yyyy-MM-dd-HH}". If it ends with ".gz" the old logs will be compressed when rolled
log4j.appender.oozie.RollingPolicy.FileNamePattern must start with the value of log4j.appender.oozie.File

Related

How to define log4j default file location

I'm setting up "log4j" logging framework to my java project, and I want to define a default path to the log4j.appender.X.File=${file.name}configuration property, if nothing to the ${file.name} argument has been specified by the comand line.
I need the solution to be dynamic:
- when I run the .jar file using -Dfile.name=C:\log\directory\path\log4j-application.log JVM argument on the comand line --> the framework should write to the path that I've specified on the argument;
- when I run the .jar file without using the JVM argument on the comand line --> the framework should write on the .jar file path;
Where should I give the default path to make my project understand where it should write the file on?
This is a part of my actual log4j.properties file:
# Root logger option, use one of the following log levels: INFO, WARN, ERROR, DEBUG, TRACE
log4j.rootLogger=DEBUG, stdout, file
[...]
# Redirect log messages to a log file, support file rolling.
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=${file.name}
log4j.appender.file.MaxFileSize=100MB
log4j.appender.file.MaxBackupIndex=3
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout=com.webratio.rtx.log.SessionAwarePattern
log4j.appender.file.layout.ConversionPattern= %d{dd-MM-yyyy HH:mm:ss} %-5p %c{1}:%L - %m%n
log4j.appender.file.Append=true
log4j.appender.file.Threshold=TRACE
With the attached settings, the app is working partly:
1. java -Dfile.name=C:\log\directory\path\log4j-application.log -jar C:\Users\martin\Desktop\Projects\Log4j\log4j_example.jar: including the JVM argument, the .jar correctly created the log file inside the already defined JVM argument path;
2. java -jar C:\Users\martin\Desktop\Projects\Log4j\log4j_example.jar: excluding the JVM argument, the .jar is not able to understand where it should write on, therefore no file has been created;
I was expecting that, on the second attempt, the app could write on the same .jar file folder, since the ${file.name} argument was not defined on the comand line.That didn't work because it seems that something is missing to assign that "default" path to write on.
I also tried setting log4j.appender.file.File=${file.name}/log4j.log on the log4j.properties file instead, giving java -Dfile.name=C:\log\directory\path\ -jar C:\Users\martin\Desktop\Projects\Log4j\log4j_example.jaras a comand and excluding the log4j.log name to the path.
Niether this attempt is working for me, since system is raising the following error:
[...]
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: \log4j.log (Access denied)
[...]
This time I was expecting that the framework could write on the same .jar file folder, since the ${file.name} argument was not defined on the comand line and atuomatically get the root of the .jar file (expecting that's the default writing folder).
I'm using the 1.2.17 log4j version.
#in properties file put like this
log4j.appender.file.File=./history.log

Location change for Logs in play framework

I have been newly working on Play framework and the version is 1.2.5. The logs for my application are stored at location where the entire application is installed. I have installed my application package in C:\app location and running it through Java wrapper.
I am confused as my log.properties file do not say anything about the stored location of log, as shown below, but still it is logging in default log folder of play.
Below is the exactly copied log.properties file for my application which is working fine. Few lines are commented but that is how I have received it.
#properties file used for log4j
log4j.rootLogger=ERROR, Console
log4j.logger.play=DEBUG
# Rolling files
#log4j.appender.Rolling=org.apache.log4j.RollingFileAppender
#log4j.appender.Rolling.File=logs/application.log
#log4j.appender.Rolling.MaxFileSize=1MB
#log4j.appender.Rolling.MaxBackupIndex=100
#log4j.appender.Rolling.layout=org.apache.log4j.PatternLayout
#log4j.appender.Rolling.layout.ConversionPattern=%d{DATE} %-5p ~ %m%n
# Console
log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
log4j.appender.Console.layout.ConversionPattern=%d{ABSOLUTE} %-5p ~ %m%n
All I want to know is how Play is storing it there, Is there any class of Play that has this location stored, and How can I configure to change the location of building logs to other drive say E:\logs?
It is responsible for the location of the log folder,
#log4j.appender.Rolling.File=logs/application.log
will generate logs in the application path with a folder logs
you can change it to the desired location E:\logs in the above line to generate logs in the E:
You will find the default settings of logging,
# Logger
# ~~~~~
section in application.conf inside your /conf directory of your play app.

Application specific log in tomcat 7 using JULI?

I'm using java system logging in tomcat 7, but no logging statements get written to the log. I've added this file to my WEB-INF/classes. The log file "new-xyz-test" gets created (so I have at least some of the config right) but its empty - no log statements get printed to it.
handlers=java.util.logging.ConsoleHandler, org.apache.juli.FileHandler
org.apache.juli.FileHandler.level=ALL
org.apache.juli.FileHandler.directory=${catalina.base}/logs
org.apache.juli.FileHandler.prefix=new-xyz-test-
java.util.logging.ConsoleHandler.level=ALL
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
com.xyz.level=ALL
com.xyz.handlers=org.apache.juli.FileHandler
To configure JULI in the web applications you need have a logging.properties file in the WEB-INF/classes directory. If you use the default handlers, you may lose messages. You need to specify a prefix for the handler in your file.
handlers=1FILE.org.apache.juli.FileHandler, java.util.logging.ConsoleHandler
.handlers=java.util.logging.ConsoleHandler
1FILE.org.apache.juli.FileHandler.level=FINEST
1FILE.org.apache.juli.FileHandler.directory=/app-logs
1FILE.org.apache.juli.FileHandler.prefix=file-1
java.util.logging.ConsoleHandler.level=FINE
java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
com.xyz.level=INFO
com.xyz.handlers=1FILE.org.apache.juli.FileHandler
com.abc.level=INFO
com.abc.handlers=java.util.logging.ConsoleHandler
A
handler prefix (e.g. 1FILE.) starts with a number, then has an arbitrary string, and ends with a period (.).
See more in Logging in Tomcat
Arguments in the JVM
If you are not running the Tomcat from the startup.sh or startup.bat, you need to specify:
The location of the general logging.properties for Tomcat (in the conf directory of Tomcat)
The manager org.apache.juli.ClassLoaderLogManager. This is important because allows you to configure
for each web application different loggin options. By default, a JVM process can only have a single configuration file.) ,
Similar to the next (I'm using eclipse):
-Djava.util.logging.config.file="C:\Users\Paul\workspaces\utils\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\conf\logging.properties" -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
By default, java.util.logging read the file that is included in the JDK or JRE, e.g.:
"C:\Software\jdk1.7.0_17\jre\lib\logging.properties"
Setting Tomcat Heap Size (JVM Heap) in Eclipse, for how to add arguments in the VM
are you sure that you write to the correct logger , i.e. Logger.getLogger("com.xyz")?
I think that you may got wrong when you wrote in logging.properties:com.xyz.level=ALL com.xyz.handlers=org.apache.juli.FileHandler in the case that you actually write to the logger Logger.getLogger(com.xyz.YourClass.class), that because in the logging properties file you should write the logger name which is in your case com.xyz.YourClass

Java Logging - where is my log file?

I'm having trouble finding my log files.
I'm using Java Logging - java.util.logging - in Eclipse 3.7.1 on Windows XP. The relevant lines of my logging.properties file are:
handlers= java.util.logging.FileHandler, java.util.logging.ConsoleHandler
.level=INFO
java.util.logging.FileHandler.pattern = %h/java%u.log
java.util.logging.FileHandler.limit = 50000
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter
As far as I can figure out, after I execute these two lines:
Logger logger = Logger.getLogger("test");
logger.logp(Level.INFO, "myClass", "myMethod", "Alcatraz");
my log file should be in C:\Documents and Settings\[My Windows ID]\javaX.log where X is an integer.
I have 5 different java.log files in that directory, java0.log through java4.log, but none of them contain my log record or even a record with today's date on it. I did some googling and found Tracing and Logging which implies that my logs should be at a different location, c:\Documents and Settings\[My Windows ID]\Application Data\Sun\Java\Deployment\log. There is one file there, named plugin5581819941091650582.log, but it is essentially empty:
<?xml version="1.0" encoding="windows-1252" standalone="no"?>
<!DOCTYPE log SYSTEM "logger.dtd">
<log>
</log>
Its creation date is last week. (I'm not sure what process created it; I certainly didn't create it explicitly.)
So where is my log file then? I can't think of anywhere else to look.
Also, does anyone know when changes to logging.properties take effect? If I changed the log level or the FileHandler.pattern, what would have to happen before my program saw the changes? Simply saving the changes in logging.properties is clearly not enough. Will I need to restart Eclipse? Or reboot the computer? Just curious. That's not nearly as big a deal to me as finding out where my log file actually is.
Where is your logging.properties file located? It should be available in the root of the classpath. As a sanity check, what does the following code print?
System.out.println(getClass().getClassLoader().getResource("logging.properties"));
If the code is in a static context, use
System.out.println(ClassName.class.getClassLoader().getResource("logging.properties"));
Location of log file can be control through logging.properties file. And it can be passed as JVM parameter ex : java -Djava.util.logging.config.file=/scratch/user/config/logging.properties
Details:
https://docs.oracle.com/cd/E23549_01/doc.1111/e14568/handler.htm
Configuring the File handler
To send logs to a file, add FileHandler to the handlers property in the logging.properties file. This will enable file logging globally.
handlers= java.util.logging.FileHandler
Configure the handler by setting the following properties:
java.util.logging.FileHandler.pattern=<home directory>/logs/oaam.log
java.util.logging.FileHandler.limit=50000
java.util.logging.FileHandler.count=1
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter
java.util.logging.FileHandler.pattern specifies the location and pattern of the output file. The default setting is your home directory.
java.util.logging.FileHandler.limit specifies, in bytes, the maximum amount that the logger writes to any one file.
java.util.logging.FileHandler.count specifies how many output files to cycle through.
java.util.logging.FileHandler.formatter specifies the java.util.logging formatter class that the file handler class uses to format the log messages. SimpleFormatter writes brief "human-readable" summaries of log records.
To instruct java to use this configuration file instead of $JDK_HOME/jre/lib/logging.properties:
java -Djava.util.logging.config.file=/scratch/user/config/logging.properties
The .log file is in your \workspace\.metadata folder. I'm using Eclipse 4.2.
The root cause of the problem the questioner is having is that his logging.properties file is not being read.
The file specified in java.util.logging.config.file is not read from the classpath. Instead it is read from the file system relative the current directory.
For example, running the following command java -Djava.util.logging.config.file=smclient-logging.properties SMMain will read the smclient-logging.properties from the current directory. Once the correct java.util.logging.config.file is read, the logs are generated as specified in the file.
It seems that the default location has changed. To find your logfile open the Java Console with your application. in there press "s". This prints out the System- and Deployment-Properties where you can find something like:
deployment.user.logdir = C:\Users\username\AppData\LocalLow\Sun\Java\Deployment\log
There you will find your logfiles.
If its null, then the file path would be your eclipse home directory. Your logging.properties file is not raken by the system, so point the properties file to the complete path as shown below then your log file will be generated in the place of directlyr where yor prefers it. -Djava.util.logging.config.file=D:\keplereclipse\keplerws\NFCInvoicingProject\WebContent\WEB-INF\logging.properties
Debug the your variable or logger.getHandlers(): just for the instances of FileHandler, and look for its private field: files
Make sure that your logger level including your log.
Make sure that your handler level including your log.
It will be sent to your home directory. If there's no that directory in your operate system, such as windows 95/98/ME, the file should be saved to the default path like C:\Windows\.
Reflect, the same as tip 1
Field[] handlerFiles = handler.getClass().getDeclaredFields();
AccessibleObject.setAccessible(handlerFiles, true);
for (Field field : handlerFiles) {
if (field.getName().equals("files")) {
File[] files = (File[]) field.get(handler);
System.out.println(Arrays.toString(files));
}
}
The log manager will be initialized during JVM startup and completed before the main method. You can also re-initialize it in main method with System.setProperty("java.util.logging.config.file", file), which will call LogManager.readConfiguration().

how to use log4j to write a file inside my project directory?

I have a log4j properties file which is creating a file inside my tomcat>bin folder but instead can it write the log file to my project's root dir? webapps>test>___?
Here is my log4j properties file contents.
#define the console appender
log4j.appender.consoleAppender = org.apache.log4j.ConsoleAppender
# now define the layout for the appender
log4j.appender.consoleAppender.layout = org.apache.log4j.PatternLayout
log4j.appender.consoleAppender.layout.ConversionPattern=%t %-5p %c{3} - %m%n
log4j.appender.rollingFile=org.apache.log4j.RollingFileAppender
log4j.appender.rollingFile.File=/test/a.log
log4j.appender.rollingFile.MaxFileSize=10MB
log4j.appender.rollingFile.MaxBackupIndex=2
log4j.appender.rollingFile.layout = org.apache.log4j.PatternLayout
log4j.appender.rollingFile.layout.ConversionPattern=%p %t %c - %m%n
# now map our console appender as a root logger, means all log messages will go to this appender
#for console printing
#log4j.rootLogger = DEBUG, consoleAppender
#for file printing
log4j.rootLogger = DEBUG, rollingFile
Try replacing this:
log4j.appender.rollingFile.File=/test/a.log
with this:
log4j.appender.rollingFile.File=../webapps/test/a.log
Note (by Stephen C) - the "../" means this solution depends on whether or not the Tomcat launch mechanism you use makes $CATALINA_HOME the current directory before the JVM that hosts Tomcat. Some do, and some don't.
The log4j configurations understand "${catalina.home}", so ...
log4j.appender.rollingFile.File=${catalina.home}/webapps/test/a.log
However, I don't think it is a good idea to put logs into the webapps tree because they are liable to be blown away if your webapp is redeployed.
Put them in ${catalina.home}/logs instead.
Or better still, put them in the distro-specific conventional place to put application logfiles; e.g. "/var/spool/..." or "/var/log/...".
Putting logfiles in standard places means there are less places for someone else (e.g. the guy who is the backup sysadmin when you are on holiday) to investigate if the file system fills up with old logfiles.

Categories