I have no idea why this is hanging. I'm trying to capture output from a process run through commons-exec, and I continue to hang. I've provided an example program to demonstrate this behavior below.
import java.io.DataInputStream;
import java.io.IOException;
import java.io.PipedInputStream;
import java.io.PipedOutputStream;
import org.apache.commons.exec.CommandLine;
import org.apache.commons.exec.DefaultExecutor;
import org.apache.commons.exec.ExecuteException;
import org.apache.commons.exec.PumpStreamHandler;
public class test {
public static void main(String[] args) {
String command = "java";
PipedOutputStream output = new PipedOutputStream();
PumpStreamHandler psh = new PumpStreamHandler(output);
CommandLine cl = CommandLine.parse(command);
DefaultExecutor exec = new DefaultExecutor();
DataInputStream is = null;
try {
is = new DataInputStream(new PipedInputStream(output));
exec.setStreamHandler(psh);
exec.execute(cl);
} catch (ExecuteException ex) {
} catch (IOException ex) {
}
System.out.println("huh?");
}
}
According to the javadoc, execute(CommandLine command) is synchronous, execute(CommandLine command, ExecuteResultHandler handler) on the other hand is asynchronous.
The command you invoked, java, produces output to its standard output stream. That stream must be pumped into an input stream by your invoking program. This does not happen in your program.
You have to read the input stream (is in your code) in a separate thread, because that is how piped streams work. Note that you must start the reading thread before calling execute().
See also Capturing large amounts of output from Apache Commons-Exec
According to your other question Streaming output with commons-exec? you expect large data, so you must use the piped streams and cannot use the simpler approach of using a ByteArrayInputStream as output. The answer you give yourself there, suffers from the same problem as your code here.
Related
I'm trying to perform communication between server and client using Genson library. I've detected the following problem: trying to send a message to the server my application stalls when genson on the server is trying to read the message.
Meanwhile, if I shutdown the client, the message is perfectly read and processed. I've thought it to be deadlock but not sure.
There is no such a problem with native Java serialization.
Here is my server:
import com.owlike.genson.Genson;
import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;
import java.util.Arrays;
public class Server {
public static void main(String[] args) throws IOException {
Genson genson = new Genson();
try (ServerSocket server = new ServerSocket(9991)) {
try (Socket socket = server.accept()) {
int[] loc = genson.deserialize(socket.getInputStream(), int[].class);
System.out.println("Server: " + Arrays.toString(loc));
genson.serialize(loc, socket.getOutputStream());
}
}
}
}
Here is the client:
import com.owlike.genson.Genson;
import java.io.IOException;
import java.net.Socket;
import java.util.Arrays;
public class Client {
public static void main(String[] args) throws IOException {
Genson genson = new Genson();
try (Socket socket = new Socket("localhost", 9991)) {
genson.serialize(new int[] {1, 2, 3}, socket.getOutputStream());
int[] loc = genson.deserialize(socket.getInputStream(), int[].class);
System.out.println("Client: " + Arrays.toString(loc));
}
}
}
I wound highly appreciate any help with this question. Thanks in advance.
Edit: This is really wierd. I've made some additional tests and here is what i get:
Additional class:
import com.owlike.genson.annotation.JsonProperty;
import java.io.Serializable;
public class Tester implements Serializable {
public static final Tester TEST = new Tester(Math.E);
private double val = Math.PI;
public Tester(#JsonProperty("val") double val) {
this.val = val;
}
public Tester() {}
public String toString() {
return "" + val;
}
}
Having written genson.serialize(Tester.TEST, socket.getOutputStream()) in the client request I have the same strange result. But having written genson.serialize(new Tester(Double.NaN), socket.getOutputStream()) the result is the expexted one.
Furthermore, if I define the only field in Tester class to be of type int[], lets say, it only works with values of null or new int[0].
In addition to that, if I'm trying to serialize and transmit int for integers in range 0..9 I observe the following behaviour: the same strange thing except that when I shutdown the client, server always shows 0 value.
Besides, for constants like Double.NaN, Double.POSITIVE_INFINITY, Integer.MAX_VALUE and similar there is nothing strange at all (everything works as expected).
For those additional tests Genson class was defined as follows:
Genson genson = new GensonBuilder()
.useMethods(false)
.setFieldFilter(VisibilityFilter.PRIVATE)
.create();
Note that there is no such issue when ser/deser to/from a file using streams:
import com.owlike.genson.Genson;
import com.owlike.genson.GensonBuilder;
import com.owlike.genson.reflect.VisibilityFilter;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
public class FileTest {
private static final String FILENAME = "test.json";
public static void main(String[] args) throws IOException {
Genson genson = new GensonBuilder()
.useMethods(false)
.setFieldFilter(VisibilityFilter.PRIVATE)
.useIndentation(true)
.create();
try (OutputStream stream = new FileOutputStream(FILENAME)) {
genson.serialize(Tester.TEST, stream);
}
try (InputStream stream = new FileInputStream(FILENAME)) {
System.out.println(genson.deserialize(stream, Tester.class));
}
}
}
Looks like it was a mistake of mine all the time. I've forgotten that socket's stream can't be closed (unless you want to close the socket too). So in this case Server tries to get as much data from InputStream as it can but it can't consume all the stream (because it is always opened and data can be sent at any time from the client). So the Server basically freezes waiting for data but there is no more data to come. As a result we have the very situation described above.
A solution would be to specify some kind of protocole to denote query size so the Server can know how much data it should consume. See this answer for more details.
Some code paths in the reading API will try to eagerly ensure that there are at least N bytes available or EOF has been reached. This happens all the time when parsing a number (that is not a constant) as you noted.
An option could be to implement a small layer that serializes to a byte array or string, gets the message length and writes it to the output, followed by the payload. On the reading side you would first read the length and then read from the stream in a while loop until you reached that length or EOF.
Then you would just pass to Genson.deseralize this in memory message.
I'm attempting to run an Access function through Java using Jacob using Application.Run (https://msdn.microsoft.com/en-us/library/office/ff193559.aspx). I am able to open and close an Access database, but not run a function. I suspect the run call actually does go through but that I have opened the file read-only (maybe? not sure I did) which then causes the Access error: Run-time error 3073: Operation must use an updatable query. The query simply appends two strings onto a test table I created, and that query works by hand, but so far not through Java.
If the error is that I've opened it read-only, how can I open it not read-only? If it's something else, how do I call a function (or a macro, either will work) using Jacob? Or you may know some other Java technique besides using Jacob, I'd take that too.
Minimum example:
Java program
import com.jacob.activeX.ActiveXComponent;
import com.jacob.com.ComThread;
import com.jacob.com.Dispatch;
import com.jacob.com.LibraryLoader;
import com.jacob.com.Variant;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
/**
*
* #author evans
*/
public class Test {
public static void main(String[] args) {
// Load library/.dll
try {
String libFile = System.getProperty("os.arch").equals("amd64") ? "jacob-1.18-x64.dll" : "jacob-1.18-x86.dll";
FileInputStream inputStream = new FileInputStream(new File(libFile));
File temporaryDll = File.createTempFile("jacob", ".dll");
try (FileOutputStream outputStream = new FileOutputStream(temporaryDll)) {
byte[] array = new byte[8192];
for (int i = inputStream.read(array); i != -1; i = inputStream.read(array)) {
outputStream.write(array, 0, i);
}
}
System.setProperty(LibraryLoader.JACOB_DLL_PATH, temporaryDll.getAbsolutePath());
LibraryLoader.loadJacobLibrary();
temporaryDll.deleteOnExit();
} catch (IOException ex) {
Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
}
// Open thread
ComThread.InitSTA(true);
// New application
ActiveXComponent ComBridge = new ActiveXComponent("Access.Application");
// Open database
Dispatch.put(ComBridge, "Visible", new Variant(true));
ComBridge.invoke("OpenCurrentDatabase", new Variant("C:/Users/evans/Documents/Book Business/Building Reports/Book Business.accdb"));
// Run function
ComBridge.invoke("Run", new Variant("Test"));
// Shutdown
ComBridge.invoke("Quit");
ComThread.quitMainSTA();
ComThread.Release();
}
}
Access query:
INSERT INTO tblTest ( Test, Test2 )
SELECT "a" AS Expr1, "B" AS Expr2;
How we can take input from the file in the Eclipse?
Just like we direct the I/O from the file from the command line.
java MyProgram < input.txt >output.txt
I am unable to direct the input.
but output directing is easy.
Just go->Run->Run->Configurations->Common
Why don't you use File instead of redirection?
Your program will have a fileName as input and then write the result in a file.
If you need necessarily use the default in you can do something like this:
System.setIn(new FileInputStream("testFile.txt"));
a sample of how it works follows:
import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
public class TestSystemIn {
public static void main(final String[] args) throws IOException {
// prepare test
FileOutputStream fos = new FileOutputStream("testFile.txt");
fos.write("testToken".getBytes());
// configure env
System.setIn(new FileInputStream("testFile.txt"));
// perform read test
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
System.out.println("read: " + br.readLine());
}
}
The output could be done in the same way using:
System.setOut(new PrintStream("testFile.txt"));
I want to write to a named pipe (already created) without blocking on the reader. My reader is another application that may go down. If the reader does go down, I want the writer application to keep writing to that named pipe. Something like a this in Java
fopen(fPath, O_NONBLOCK)
So that when the reader comes up, it may resume from where it failed.
First I try to answer your questions. Next I will try to show you a code snippet I created that solves your problem using blocking IO.
Your questions
I want to write to a named pipe
(already created) without blocking on
the reader
You don't need non blocking IO to solve your problem. I think it can not even help you solve your problem. Blocking IO will also run good(maybe even better then non blocking IO because of the low concurrency). A plus is blocking IO is easier to program. Your reader can/should stay blocking.
My reader is another application that
may go down. If the reader does go
down, I want the writer application to
neep writing to the named pipe. So that when the reader comes up, it may resume from where it failed.
just put the messages inside a blocking queue. Next write to the named pipe only when the reader is reading from it(happens automatically because of blocking IO). No need for non-blocking file IO when you use a blocking queue. The data is asynchronous delivered from the blocking queue when a reader is reading, which will sent your data from your writer to the reader.
Something like a fopen(fPath,
O_NONBLOCK) in Java
You don't need non-blocking IO on the reader and even if you used it. just use blocking IO.
CODE SNIPPET
A created a little snippet which I believe demonstrates what your needs.
Components:
Writer.java: reads lines from console as an example. When you start program enter text followed by enter which will sent it to your named pipe. The writer will resume writing if necessary.
Reader.java: reads lines written from your named pipe(Writer.java).
Named pipe: I assume you have created a pipe named "pipe" in the same directory.
Writer.java
import java.io.BufferedWriter;
import java.io.Console;
import java.io.FileWriter;
import java.io.PrintWriter;
import java.util.concurrent.BlockingDeque;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingDeque;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Writer {
private final BlockingDeque<StringBuffer> queue;
private final String filename;
public static void main(String[] args) throws Exception {
final Console console = System.console();
final Writer writer = new Writer("pipe");
writer.init();
while(true) {
String readLine = console.readLine();
writer.write(new StringBuffer(readLine));
}
}
public Writer(final String filename){
this.queue = new LinkedBlockingDeque<StringBuffer>();
this.filename = filename;
}
public void write(StringBuffer buf) {
queue.add(buf);
}
public void init() {
ExecutorService single = Executors.newSingleThreadExecutor();
Runnable runnable = new Runnable() {
public void run() {
while(true) {
PrintWriter w = null;
try {
String toString = queue.take().toString();
w = new PrintWriter(new BufferedWriter(new FileWriter(filename)), true);
w.println(toString);
} catch (Exception ex) {
Logger.getLogger(Writer.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
};
single.submit(runnable);
}
}
Reader.java
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Reader {
private final BufferedReader br;
public Reader(final String filename) throws FileNotFoundException {
br = new BufferedReader(new FileReader(filename));
}
public String readLine() throws IOException {
return br.readLine();
}
public void close() {
try {
br.close();
} catch (IOException ex) {
Logger.getLogger(Reader.class.getName()).log(Level.SEVERE, null, ex);
}
}
public static void main(String[] args) throws FileNotFoundException {
Reader reader = new Reader("pipe");
while(true) {
try {
String readLine = reader.readLine();
System.out.println("readLine = " + readLine);
} catch (IOException ex) {
reader.close();
break;
}
}
}
}
If you want pipes to stay active and queue up messages, you probably want a messaging system rather than a raw pipe. In Java, the standard API is called "Java Messaging System" (JMS), and there are many standard implementations-- the most common of which I've seen being Apache ActiveMQ. If you want a cross-platform, sockets-like interface that does buffering and recovery I might suggest 0MQ, which while not being "pure Java" has bindings for many languages and excellent performance.
If there was such a thing as non-blocking file I/O in Java, which there isn't, a write to a named pipe that wasn't being read would return zero and not write anything. So non-blocking isn't part of the solution.
There's also the issue that named pipes have a finite buffer size. They aren't infinite queues regardless of whether there is a reading process or not. I agree with the suggestion to look into JMS.
You should be able to use NIO's asynch write on a UNIX FIFO, just as you can to any other file:
AsynchronousFileChannel channel = AsynchronousFileChannel.open(...);
Future<Integer> writeFuture = channel.write(...);
... or...
channel.write(..., myCompletionHandler);
However, it's not clear to me what you want to happen when the FIFO isn't accepting writes. Do you want it to buffer? If so you'll need to provide it within the Java program. Do you want it to time out? There's no simple timeout option on Java file writes.
These aren't insurmountable problems. If you're determined you can probably get something working. But I wonder whether you'd not find life much easier if you just used a TCP socket or a JMS queue.
So I am trying to write an .sh file that will be executable, this is how I'm currently writing it:
Writer output = null;
try {
output = new BufferedWriter(new FileWriter(file2));
output.write(shellScriptContent);
output.close();
} catch (IOException ex) {
Logger.getLogger(PunchGUI.class.getName()).log(Level.SEVERE, null, ex);
}
So that writes the file just fine, but it is not executable. Is there a way to change the executable status when I write it?
Edit: To further clarify, I am trying to make it execute by default, so that for instance, if you double clicked the generated file, it would automatically execute.
You can call File.setExecutable() to set the owner's executable bit for the file, which might be sufficient for your case. Or you can just chmod it yourself with a system call with Process.
Alas, full-powered programmatic alteration of file permissions isn't available until Java 7. It'll be part of the New IO feature set, which you can read more about here.
You'd need to chmod it, and you can probably do it by exec'ing a system command like such:
Really all you'd need is to fire off something like this:
Runtime.getRuntime().exec("chmod u+x "+FILENAME);
But if you want to keep track of it more explicitly can capture stdin / stderr then something more like:
Process p = Runtime.getRuntime().exec("chmod u+x "+FILENAME);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(p.getErrorStream()));
Which I got from here:
http://www.devdaily.com/java/edu/pj/pj010016/pj010016.shtml
Update:
Test program:
package junk;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Writer;
public class Main{
private String scriptContent = '#!/bin/bash \n echo "yeah toast!" > /tmp/toast.txt';
public void doIt(){
try{
Writer output = new BufferedWriter(new FileWriter("/tmp/toast.sh"));
output.write(scriptContent);
output.close();
Runtime.getRuntime().exec("chmod u+x /tmp/toast.sh");
}catch (IOException ex){}
}
public static void main(String[] args){
Main m = new Main();
m.doIt();
}
}
On linux if you open up a file browser and double click on /tmp/toast.sh and choose to run it, it should generate a text file /tmp/toast.txt with the words 'yeah toast'. I assume Mac would do the same since it's BSD under the hood.
In Java 7 you can call Files.setPosixFilePermissions. Here is an example:
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.attribute.PosixFilePermission;
import java.util.Set;
class FilePermissionExample {
public static void main(String[] args) throws IOException {
final Path filepath = Paths.get("path", "to", "file.txt");
final Set<PosixFilePermission> permissions = Files.getPosixFilePermissions(filepath);
permissions.add(PosixFilePermission.OWNER_EXECUTE);
Files.setPosixFilePermissions(filepath, permissions);
}
}
On Mac OS X, besides chmod +x, you have to give a .command extension to your shell script if you want to launch it with a double-click.
This answer I wrote for the question how do I programmatically change file permissions shows a chmod example via a native call using jna, which should work on Mac OS X.