We Keep Coding

Java console log simultaneously to readLine without collisions

I'm currently dealing with the following problem:
I try to make a console input for a java application
that works with multiple threads. So while running the
software it happens sometimes, that a new line of log is
appearing while I'm writing to the readLine with a promt..
When that happens it looks like the following:
Image of the Console
so it does stack the messages like in the image.. so here is the question:
How can I keep the line and text I am writing to and log the text above it like in the example below?
Gif of the input
(Sorry for low quallity but you can guess what I mean)
As you can see my input stays at the bottom, is still editable and the lines do not stack
Thank you for some help, I am struggeling so much after trying Log4j, System.console, BufferedReaders and Scanner

Solved. It was horrible complicated..
But here for the future:
class ConsoleThread implements Runnable {
private ConsoleReader reader;
private OutputStream output;
public ConsoleThread(OutputStream output, ConsoleReader reader) {
this.output = output;
this.reader = reader;
}
#Override
public void run() {
try {
String message;
while (true) {
message = LoggingQueue.getNextLogEvent();
if(message == null) continue;
reader.print(Ansi.ansi().eraseLine(Ansi.Erase.ALL).toString() + ConsoleReader.RESET_LINE);
reader.flush();
output.write((message + System.lineSeparator()).getBytes());
output.flush();
try {
reader.drawLine();
} catch (Throwable ex) {
reader.getCursorBuffer().clear();
}
reader.flush();
}
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
}
}
Using the ConsoleReader of jLine2 and jAnsi. The output stream is just System.out.
You just need a second thread which reads and you are done :)

You need to handle the threads competing for stdin/stdout.
In theory, that would imply some kind of mutex but, since you're using external libraries, it seems like too much trouble...
From your images, it seems that you're running a sort of server application that takes commands.
If that's the case, I recommend re-architecting to use two separate processes: one for the server part and one for the command prompt.
The two processes then communicate through a socket.
This allows you to make the command prompt single threaded or, at least, behave like a single threaded application, since it is only reacting to user commands.
This is what lots of applications, like Docker, Kubernetes or MySQL do.
In the case of Docker and Kubernetes, they expose full REST APIs on that socket so you can leverage libraries for that.

Previous pdf in response

I have servlet, which create html-file and then convert it to pdf-file:
private void ConvertHTMLtoPDF(String sConvertationProgramm, String sHTML, String sPDF)
{
try {
ProcessBuilder pb = new ProcessBuilder(sConvertationProgramm, sHTML, sPDF);
Process process = pb.start();
} catch (Exception e) {
e.printStackTrace();
}
}
Everything work perfect, but then I open this new (as I think) generated pdf-file from this code:
ConvertHTMLtoPDF("C:\\Program Files (x86)\\wkhtmltopdf\\wkhtmltopdf.exe",
"PDFtemplate/requiredPDF.html",
"PDFtemplate/Report.pdf");
response.sendRedirect("PDFtemplate/Report.pdf");
- that gave me previous pdf-file, which was created before.
Furthermore I've tried to open it from Windows explorer, and it's also showed me previous file every first opens.
Does anybody have any suggestion what's happen and how to solve this problem?
Any input would be greatly appreciated,
Thanks

You are doing response.sendRedirect("PDFtemplate/Report.pdf") after starting some process. You should wait finish of that process and only after finishing make other actions.
Use process.waitFor();

Are you sure you are replacing your current file with the new one, I doubt you are writing your new file into this location.

Embedded Java app crashes, no exception and no apparent reason

I have a Java application running on an embedded linux (BusyBox v1.12.4). I am using CDC 1.1 and the version of the VM (cvm) is CDC HI phoneme_advanced-Core-1.1.2-b111.
The application's main purpose is to collect some data and send it via GPRS (using the FTPClient of the Apache commons library)
The application was running fine, and then lately I added the ability to compress a file before sending it. Following is the code that compresses the file:
public static boolean compressFile(String file, String fileCompressed)
{
boolean result = false;
try
{
Process process = Runtime.getRuntime().exec("tar -czvf " + fileCompressed + " " + file);
System.err.println("Compression in progress");
int returnValue = process.waitFor();
System.err.println("Finished compression");
BufferedReader stderror = new BufferedReader(new InputStreamReader(process.getErrorStream()));
String s;
s = stderror.readLine();
if (s == null)
{
result = true;
} else
{
result = false;
System.err.println(s);
}
} catch (IOException e)
{
result = false;
Log.getInstance().newMessage(e.getMessage(), Log.ERROR);
} catch (InterruptedException e)
{
result = false;
Log.getInstance().newMessage(e.getMessage(), Log.ERROR);
}
return result;
}
After adding this function, the application started crashing! The log didn't contain any memory error or exception and the system it's running on lacks correct configuration, so the syslog doesn't show anything as well (the equipment maker told me that it will be available in the upcoming version). I can't even launch the VM in debug mode!
I have to add that the application doesn't crash during the compression, it just crashes randomly during its execution. The thing is that it crashes only when compression is enabled!
Has anybody seen this before? does anybody have an idea on how to debug/solve this?

The first thing I would suspect would be an out-of-memory condition, caused by the execution of the gzip compression engine. Try calling dmesg in the shell to see if the kernel OOM killer sacrificed any processes.
A few other things to try:
Remove the v flag of tar: You do not seem to need the extra output and it might cause some kind of problem down the road, depending on you exact environment.
Use tar without the z option: If the crashes go away with compression disabled, then it's probably a memory issue.
Try cat, one of the the simplest external processes that you could start, instead of tar and see what happens.
Stress-test your hardware: using compression may simply be exposing underlying overheating or data corruption issues that are not normally visible.

I'd start by looking at the exit code of the JVM -- that'll tell you if it died from a signal, and if so, which.

Running a command line program multiple times in Java - is this correct?

I have a program (in Java) that needs to use another program multiple times, with different arguments, during it's execution. It is multi-threaded, and also needs to do other things besides calling that program during it's execution, so I need to use Java to do that.
The problem is, all Runtime.exec() calls seem to be done in a synchronized way by Java, such that threads get bottlenecked not around the functions themselves, but in the Java call. Thus, we have a very slow running program, but that does not bottleneck at any system resource.
In order to fix that problem, I decided to not close the Process, and make all calls using this script:
#!/bin/bash
read choice
while [ "$choice" != "end" ]
do
$choice
read choice
done
And all the previous exec calls are substituted by this:
private Process ntpProc;
Initializer(){
try {
ntpProc = Runtime.getRuntime().exec("./runscript.sh");
} catch (Exception ex) {
//Error Processing
}
}
public String callFunction(String function) throws Exception e{
OutputStream os = ntpProc.getOutputStream();
String result = "";
os.write((function + "\n").getBytes());
os.flush();
BufferedReader bis = new BufferedReader(new InputStreamReader(ntpProc.getInputStream()));
int timeout = 5;
while(!bis.ready() && timeout > 0){
try{
sleep(1000);
timeout--;
}
catch (InterruptedException e) {}
}
if(bis.ready()){
while(bis.ready()) result += bis.readLine() + "\n";
String errorStream = "";
BufferedReader bes = new BufferedReader(new InputStreamReader(ntpProc.getErrorStream()));
while(bes.ready()) errorStream += bes.readLine() + "\n";
}
return result;
}
public void Destroyer() throws exception{
BufferedOutputStream os = (BufferedOutputStream) ntpProc.getOutputStream();
os.write(("end\n").getBytes());
os.close();
ntpProc.destroy();
}
That works very well, and actually managed to improve my program performance tenfold. SO, my question is: Is this correct? Or am I missing somethings about doing things this way that will make everything go terribly wrong eventually?

If you want to read from the process Error and Input streams ( aka stderr and stdout ), you need to do this job on dedicated threads.
The main problem is that you need to empty the buffers as they become filled up, and you can only do this on a separate thread.
What you did, you've managed to shorten the output, so it does not overflow these buffers, but the underlying problem is still there.
Also, from the past experience, calling external process from Java is extremely slow, so your approach may be better after all.

As long as you not calling Proccess.waitFor(), execution of process will not block. As Alex said - blocking in your case caused by those loops to read the output.
You can use commons-exec package, as it provides a nice way of running processes (sync or async), handling output, setting timeouts, etc.
Here is a link to a project:
http://commons.apache.org/exec/
The best example of using the api is test class they have:
http://svn.apache.org/viewvc/commons/proper/exec/trunk/src/test/java/org/apache/commons/exec/DefaultExecutorTest.java?view=markup

java process is frozen(?) on linux

This is my first question on S.O.
I have a very odd problem.
Below is my problem...
I write very simple method that write some text to a file.
Of course it works well my machine(XP, 4CPU, jdk1.5.0_17[SUN])
But it somtimes freezes on operating server
(Linux Accounting240 2.4.20-8smp, 4CPU, jdk1.5.0_22[SUN]).
kill -3 doesn't work.
ctrl + \ doesn't work.
So, I can't show you the thread dump.
It freezes well..
When I just write some Thread.sleep(XX) at this method, the problem is gone well(?)...
sleep(XX) break... it happened again today with Thread.sleep(XX)...
Do you know this problem?
Do you have the some solution about that?
Thanks. :-)
P.S.
linux distribution: Red Hat Linux 3.2.2-5
command: java -cp . T
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileOutputStream;
import java.io.OutputStreamWriter;
import java.text.SimpleDateFormat;
import java.util.Date;
public class T {
private BufferedWriter writer = null;
private void log(String log) {
try {
if (writer == null) {
File logFile = new File("test.log");
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream(logFile, true)));
}
writer.write(new SimpleDateFormat("[yyyy-MM-dd HH:mm:ss] ")
.format(new Date()));
writer.write("[" + log + "]" + "\n");
writer.flush();
/*
* this is ad hoc solution ???
*/
//Thread.sleep(10);
} catch (Exception e) {
e.printStackTrace();
} finally {
}
}
public void test() {
long startTime = System.currentTimeMillis();
while (true) {
log(String.valueOf(System.currentTimeMillis()));
System.out.println(System.currentTimeMillis());
try {
//Thread.sleep((int) (Math.random() * 100));
} catch (Exception e) {
break;
}
if (System.currentTimeMillis() - startTime > 1000 * 5) {
break;
}
}
if (writer != null) {
try {
writer.close();
} catch (Exception e) {
}
}
System.out.println("OK");
}
public static void main(String[] args) {
new T().test();
}
}

If the JVM does not respond to kill -3 then it is not your program but the JVM that is failing which is bad and would require a bug report to Sun.
I noticed you are running a 2.4.20-8smp kernel. This is not a typical kernel for a current open source Linux distribution, so I would suggest you have a look at http://java.sun.com/j2se/1.5.0/system-configurations.html to see if you are deploying to a supported configuration. If not, you should let the responsible people know this!

The first step is to get a thread dump of where the program is when it "freezes". If this were on Java 6, you could connect JVisualVM or JConsole to it by default, and get the stacktraces of all the threads from there. Since it's Java 5, you should be able to use the jstack command to get a thread dump (or you could enable JMX with a command-line option to attach the aforementioned tools, but I don't think it's worth it in this case). In all cases, pressing Ctrl-Break from the console that launched the application may also produce a thread dump, depending on the environment.
Do this several times a few seconds apart and then compare the thread dumps. If they're always identical, then it looks like your application is deadlocked; and the top line of the dump will show exactly where the threads are blocking (which will give a very good clue, when you look at that line of the code, which resources they're blocked on).
On the other hand if the thread dumps change from time to time, the program is not strictly deadlocked but looks like it's running in an infinite loop - perhaps one of your loop conditions is not declared properly so the threads never exit or something of that sort. Again, look at the set of thread dumps to see what area of code each thread is looping around in, which will give you an idea of the loop condition that is never evaluating to an exit condition.
If the issue isn't obvious from this analysis, post back the dumps as it will help people debug your above code.

I think this is a race condition. The while(true) will force the VM on linux to write and flush continuously, and the linux kernel VM will try to intercept those calls and buffer the writing. This will make the process spinloop while waiting for the syscall to be completed; at the same time, it will be picked up by the scheduler and assigned to another CPU (I might be wrong here, tho). The new CPU will try to acquire a lock on the resource, and everything will result in a deadlock.
This might be a sign of other issues to come. I suggest:
first of all, for clarity's sake: move the file creation outside of the log() method. That's what constructors are for.
secondly, why are you trying to write to a file like that? Are you sure your program logic makes sense in the first place? Would you not rather write your log messages to a container (say, an ArrayList) and every XX seconds dump that to disk in a separate thread? Right now you're limiting your logging ability to your disk speed: something you might want to avoid.