Every once in a while our catalina.out file gets very very large (yes, I will be implementing slf4j and logback in my applications to prevent this in the future). But for now, when I go to cycle the logs, I copy catalina.out to catalina.{date} and execute cat /dev/null > catalina.out. The problem is, tomcat will capture no further logs after I do that, until tomcat is restarted the next morning, and this is not ideal. Why does this happen? And is there a way to avoid it?
Easy as cake: echo > catalina.out. The file descriptor won't change and java can continue to write to that file.
The traditional way is to
cat /dev/null > catalina.out
It will clear the log file, and will not disrupt the processes that currently hold open file handles.
The better way is to not lose your logging information, by rotating out the log file. To do this, create or edit the file /etc/logrotate.d/tomcat and have its contents read
/var/log/tomcat/catalina.out { copytruncate daily rotate 7 compress missingok size 5M }
Then restart logrotate with the command (as root)
/usr/sbin/logrotate /etc/logrotate.conf
And you should have the log file rotated out daily, or if the size exceeds 5M, with the last seven logs kept for debugging purposes.
You can truncate the file. This makes logical sense also since it is essentially what you are trying to do.
truncate -s 0 M catalina.out
FYI: Doing a cat /dev/null > file does not alter the inode of the file.
logs]$ls -i test.log
19794063 test.log
logs]$
logs]$cat /dev/null > test.log
logs]$ls -i test.log
19794063 test.log
Also, I had a separate command tailing live data into test.log during these commands. After running these commands the tail to test.log still worked without a problem. This does not answer your question as to why it stopped working, but it helps rule out change in inode.
What you are looking for is log rotation. Since this requires a low-level signal support while the process is running, you should do this indirectly. You can use this procedure to achieve this. This is also documented on Tomcat Wiki.
go to the folder /opt/tomcat/logs/
type in command line
sudo echo > catalina.out
without stopping tomcat. . this will always work .
you can also schedule this command on daily or weekly basis . .
go to the folder /opt/tomcat/logs/
type in command line --
sudo echo > catalina.out
you can run this without shutting down tomcat server . .
you can also schedule this command on daily or weekly basis . .
If the catalina.out is deleted after tomcat is stopped, it will create a new catalina.out once tomcat starts again and it is totally safe.
But if you remove the catalina.out while tomcat is running, it will keep on logging to catalina.out which is removed already (reference of the file is hold by the tomcat) hence the space will not be released. So you will need to restart the tomcat sever to release the space. It is not recommended.
Actually you can clear the log of catalina.out file without stopping tomcat with the command below.
sudo cat /dev/null > /opt/tomcat/apache-tomcat-9.0.37/logs/catalina.out
To keep catalina.out smaller, either one of the following ways can be used:
You could use the above Linux command and add it to a CronScheduler
to run daily/ weekly/ monthly or yearly as preferred.
Use logrotate tool in Linux. It is a log managing command-line tool
in Linux. This can rotate the log files under different conditions.
Particularly, we can rotate log files on a fixed duration or if the
file has grown to a certain size. You can click here for more
info on this.
Simple and straight forward solution:
echo "" > catalina.out
Note: Add sudo if required
Related
I am fairly new to Amazon. I have a Java file which reads GBs of crawled data and I am running this using AWS ToolKit for Eclipse. The disadvantage here is, I have to keep my machine running for weeks if I need to read the entire crawled data and that is not possible. Apart from that, I can't download GBs of data in to my local PC (Because it is reading data).
Is there any way that I can upload the Jar to Amazon, and Amazon run it without engaging with my computer? I have heard about web crawlers running in Amazon for weeks without downloading data into the developers machine, and without letting the developer to turn on his machine without shutting down for months.
The feature I am asking is just like "job flows" in Amazon Elastic Map-Reduce. You upload the code, it runs it inside. It doesn't matter whether you keep "your" machine turned on or not.
You can run with the nohup command for *nix
nohup java -jar myjar.jar 2>&1 >> logfile.log &
This will run your jar file, directing the output [stderr and stdout] to logfile.log. The & is needed so that it runs in the background, freeing up the command line / shell/
!! EDIT !!
It's worth noting that the easiest way I've found for stopping the job once it's started is:
ps -ef | grep java
Returns ec2-user 19082 19056 98 18:12 pts/0 00:00:11 java -jar myjar.jar
Then kill 19082.
Note, you can tail -f logfile.log or other such derivatives [less, cat, head] to view the output from the jar.
Answer to question/comment
Hi. You can use System.out.println(), yes, and that'll end up in logfile.log. The command that indicates that is the 2&>1 which means "redirect stream 2 into stream 1". In unix speak that means redirect/pipe stderr into stdout. We then specify >> logfile.log which means "append output to logfile.log". As System.out.println() writes to stdout it'll end up in logfile.log.
However, if you're app is set up to use Log4j/commons-logging then using LOG.info("statement"); will end up in the configured 'log4j.properties' log file. With this configuration the only statements that will end up in logfile.log will be those that are either System generated (errors, linux internal system messages) or anything that's written explicitly to the stdout (ie System.out.println()) statements;
I noticed a lot of file /tmp/.java_pid<...> in my Linux machine. The file command says they are sockets. Assuming they are created by Java I wonder why Java does not clean them up. How to make Java clean them up or just not create them?
These files are created by the JVM to support debugging. It's part of the attach api.
If you don't want java to create them then start java apps without debugging enabled.
You can safely delete them if there isn't a jvm with the corresponding pid... a task that is eminently suitable for a cron job.
A little bit of bash:
for file in /tmp/.java-[0-9]*; do
[ -d /proc/${file#*.java-} ] || rm -f $file
done
pid files are generally the location where applications store their process id, so the user can kill the process easily afterwards. These applications should be deleting these files when they close down.
I wouldn't worry about these files too much, unless you are seeing more and more of them and they dont get deleted, then it might be a tell tale sign that you have an application that is not shutting down correctly,
I'm trying to build a script that starts minecraft_server.jar (location: user directory / mineserver / minecraft_server.jar ).
I have PHP and Apache installed, and I'm trying trying to start the server JAR from /var/www/html/interface.php.
Via console, the server starts fine by running:
java -Xmx1024M -Xms1024M -jar mineserver/minecraft_server.jar nogui
... from the user directory. So in the interface.php file I have the following (note location listed above):
system('java -Xmx1024M -Xms1024M -jar /home/ec2-user/mineserver/minecraft_server.jar nogui', $retval);
But the server never starts after I visit the file. What am I doing wrong?
Thanks for any and all clues.
I want to say thank you for your help on Thanksgiving. While I haven't found the solution yet, I appreciate the efforts. Thanks again.
Have you checked the error log? It's at /var/log/httpd by default. If anything went wrong, it should be there. Otherwise, it could be starting wrong and I would recommend launching it in a screen so you can hop in and check on anything it's doing.
Install screen through whatever method your distribution uses, and then change the launch string to be:
screen -dmS "minecraft" java -Xmx1024M -Xms1024M -jar /home/ec2-user/mineserver/minecraft_server.jar nogui
The command will be executed with the permissions, and the environment of whatever user account the web server runs under.
In particular, it may have a different PATH entry, and the java command might not even be found, or it might be found somewhere else etc...
Also, be aware of what current working directory it is inheriting from your PHP script.
See exec() and check the output to help yourself do some basic debugging.
Also, you probably want something more like
exec('nohup java etc... > /dev/null 2>&1 &');
so that the process gets properly backgrounded and disconnected from the webserver parent process.
So I have the following problem: I have a web service running inside a Tomcat7 server on Linux. The web service however has to execute some commands (mostly file operations such as copy and mount). Copy I've replaced with java.nio, but I don't think that there is a replacement for mount.
So I'm trying to execute shell commands out of my Tomcat Java process. Unfortunately it doesn't execute my commands. I've implemented the execution of shell commands in Java before. So my code should be correct:
Process pr = Runtime.getRuntime().exec("mount -o loop -t iso9660 <myimage> <mymountpoint>");
pr.waitFor();
<myimage> and <mymountpoint> are absolute paths, so no issues there either.
I've debugged my commands and they are working when executed on the console.
I've tried sending other commands. Simple commands such as id and pwd are working!
I've tried using /bin/bash -c "<command>", which didn't work.
I've tried executing a shell script, which executes the command, which didn't work.
I've tried escaping the spaces in my command, which didn't work.
So I've digged even deeper and now I'm suspecting some Tomcat security policy (Sandbox?), which prevents me from executing the command. Since security is no issue for me (it's an internal system, completely isolated from the outside world), I've tried a hack, which became quite popular just recently:
System.setSecurityManager(null);
This didn't work either. I'm using Java7 and Tomcat7 on RHEL6. Tomcat7 is just extracted! I don't have any files in /etc/.. or any other folder than /opt/tomcat/, where I've extracted the zip from the Tomcat home page. I've searched the /opt/tomcat/conf folder for security settings, but all I could find was the file catalina.policy, where it didn't seem like I could set some security level for shell commands.
Any ideas?
A few things:
System.setSecurityManager(null);
you have just killed the security of your application.
Yes, Tomcat is running as root. If I execute id I'm root as well.
Fix this immediately!
Now on to the question. You shouldnt have Tomcat executing anything, you need to defer this to a separate process whether that be a shell script or another Java program. This should also remove what (I hope) was a dependency on root running Tomcat. It should be possible to perform this command as a non-privileged user that cannot log into the system normally. You would do this by configuring /etc/fstab and supplying that same user the permissions to do this. From a pure security POV the process that mounts should not be owned by the tomcat user. Nor should the tomcat user ever be root. So to recap:
1) Stop running Tomcat as root
2) Create a separate process outside of the context of Tomcat to run this mount
3) Create a tomcat user, this user should not be able to log into the system nor should it be a privileged user (admin,super user, etc)
4) Create a process user, this user should be configured exactly as the tomcat user
5) Edit /etc/fstab giving the process user the necessary permissions to mount correctly.
It's generally a bad idea to use the single-string form of Runtime.exec. A better option is to use ProcessBuilder, and split up the arguments yourself rather than relying on Java to split them for you (which it does very naïvely).
ProcessBuilder pb = new ProcessBuilder("/bin/mount", "-o", "loop", /*...*/);
pb.redirectErrorStream(true); // equivalent of 2>&1
Process p = pb.start();
You say you're on RHEL so do you have selinux active? Check your logs and see if this is what's blocking you (I think it's audit.log you're looking for, it's been a few years since I've used selinux). If this does turn out to be the problem then you should probably ask on superuser or serverfault rather than SO...
I'm not sure if that's the problem you are having, but I've seen issues when Runtime.exec() is used without reading the associated output buffers. You can find a detailed explanation and potential solutions here. Reading the output and error streams can also help you figure out what's going on at the OS level when you run the command.
I've recently had to do something like this from a Swing app.
You'll probably be able to pull it off with ProcessBuilder, as in Ian's answer, but I found that once things start to get complex, it's easier to write a shell script that does what you want, enabling you to pass as few parameters as possible. Then use ProcessBuilder to invoke the shell script.
If you're invoking anything that has more than really minimal output, you'll also have to read the output and error streams to keep the process from blocking when the output buffers fill, as it seems you are already doing.
I use sudo -S before command and for the tomcat7 user: tomcat7 ALL=(ALL) NOPASSWD:ALL
I used to generate thread dumps by running kill -quit and I would get them in a log file where my server logs were there. When the file grew too large I removed it using rm and created a new file of the same name.
Now when I use kill -quit to take the thread dumps, nothing gets copied in the log file - its empty.
Can anyone help?
The default JBoss startup scripts on Unix usually look something like:
nohup $JBOSS_HOME/bin/run.sh $JBOSS_OPTS >/dev/null 2>&1 &
This is unfortunate because it's sends stderr to /dev/null. Usually this is not a problem, because once log4j initializes, then most application output will go to boot.log or server.log. However, for thread dumps, and other low level errors they get lost.
Your best bet is to change the startup script to redirect stdout and stderr to a file. Additionally, one thing that's overlooked in the default setup is redirect stdin. For daemon processes it's a best practice to redirect stdin to /dev/null. For example:
nohup $JBOSS_HOME/bin/run.sh $JBOSS_OPTS >> console-$(date +%Y%m%d).out 2>&1 < /dev/null &
Lastly, if you have a running process, you can use jstack, which is included with the JRE, to get a thread dump. This will output to the console from which it's invoked. I prefer the output from kill -3, but jstack also allows you to view native stack frames.
If this is on *nix, when you delete a file, everyone who has that file still open will continue to write to the old (now missing) file. The file will only be really deleted when all file handles to it are closed.
You would have to cause the JVM to close and re-open the log file. Not sure if this can be done without a restart.
If you go into jmx and find jboss.system:service=Logging,type=Log4jService you can then invoke the reconfigure method which should cause log4j to reopen any of its log files. Then the kill -quit should work.