We Keep Coding

Py4j launch_gateway not connecting properly

I am trying to use py4j to open up a gateway that I can use to pass objects from java into python. When I try to open a gateway with the py4j function launch_gateway it does not seem to properly connect to my Java class. However, when I launch my java class in the command line and then connect to it in python using JavaGateway everything works as expected. I would like to be able to use the built in method as I am sure that I am not accounting for things that have already been considered in the design of py4j, but I'm just not sure what I'm doing wrong.
Let's say I wanted to create a gateway to the class sandbox.demo.solver.UtilityReporterEntryPoint.class. In the command line I can do this by executing the following:
java -cp /Users/grr/anaconda/share/py4j/py4j0.10.4.jar: sandbox.demo.solver.UtilityReporterEntryPoint py4j.GatewayServer
This launches as expected and I can use the methods in my class from within python after connecting to the gateway. So far so good.
My understanding of the py4j documentation would lead me to believe I should do the following to launch the gateway in python:
port = launch_gateway(classpath='sandbox.demo.solver.UtilityReporterEntryPoint')
params = GatewayParameters(port=port)
gateway= JavaGateway(gateway_parameters=params)
I get no errors when executing these three lines, but when I try to access my java class methods with gateway.entry_point.someMethod() it fails with the following error:
Py4JError: An error occurred while calling t.getReport. Trace:
py4j.Py4JException: Target Object ID does not exist for this gateway :t
at py4j.Gateway.invoke(Gateway.java:277)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)
Obviously something is not getting called correctly within launch_gateway or I am feeding it the wrong information.
In the py4j source code for launch_gateway you can see that given the inputs you provide and those constructed by the function, a command is constructed that eventually gets called by subprocess.Popen. So given the input passed to launch_gateway above the command passed into Popen would be:
command = ['java', '-classpath', '/Users/grr/anaconda/share/py4j/py4j0.10.4.jar:sandbox.demo.solver.UtilityReporterEntryPoint', 'py4j.GatewayServer', '0']
Passing this command to Popen returns the listening port as expected. However, connecting to this listening port still does not allow access to my class methods.
Finally, passing the command as a single string to Popen without the final argument ('0'), properly launches a gateway which again operates as expected. Having taken a glance at the Java source code for py4j.GatewayServer.class this makes no sense as the main method seems to indicate that the class should exit with status 1 if the length of arguments is 0.
At this point I'm kind of at a loss. I can hack my way into a workable solution, but as I said I'm sure that ignores important aspects of the gateway behavior and I don't like hacky solutions. I'd love to tag #Barthelemy in this one, but hopefully he reads this. Thanks in advance for any help.
EDIT
For now I have been able to work around this issue with the following steps.
Package entire project including all external dependencies into a single jar file magABM-all.jar, with 'Main-Class' set to UtilityReporterEntryPoint.
Include if...else block regarding presence of --die-on-exit exactly like it is in GatewayServer.java
Use subprocess.Popen to call the command to run the project jar.
UtilityReporterEntryPoint.java
public static void main(String[] args) throws IOException {
GatewayServer server = new GatewayServer(new UtilityReporterEntryPoint());
System.out.println("Gateway Server Started");
server.start();
if (args[0].equals("--die-on-exit")) {
try {
BufferedReader stdin = new BufferedReader(new InputStreamReader(System.in, Charset.forName("UTF-8")));
stdin.readLine();
System.exit(0);
} catch (java.io.IOException e) {
System.exit(1);
}
}
}
app.py
def setup_gateway()
"""Launch a py4j gateway using UtilityReporterEntryPoint."""
process = subprocess.Popen('java -jar magABM-all.jar --die-on-exit', shell=True)
time.sleep(0.5)
gateway = JavaGateway()
return gateway
In this way I can still use gateway.shutdown if necessary and if the python process that starts the py4j gateway dies or is closed the gateway will be closed.
N.B I would by no means consider this a final solution as py4j was written by much smarter individuals with a clear purpose in mind and I am sure that there is a way to manage this exact workflow within the confines of py4j. This is just a stopgap solution.

There are a few issues:
The classpath parameter in launch_gateway should be a directory or a jar file, not a class name. For example, if you want to include additional Java libraries, you would add them to the classpath parameter.
The error you receive when you call gateway.entry_point.someMethod() means that you have no entry point. When you call launch_gateway, the JVM is started with GatewayServer.main, which launches a GatewayServer with no entry point: GatewayServer server = new GatewayServer(null, port). It is not possible currently to use launch_gateway and specify an entry point.
When you start the JVM with java -cp /Users/grr/anaconda/share/py4j/py4j0.10.4.jar: sandbox.demo.solver.UtilityReporterEntryPoint py4j.GatewayServer I believe the JVM uses UtilityReporterEntryPoint as the main class. Although you did not provide the code, I assume that this class has a main method and that it launches a GatewayServer with an instance of UtilityReporterEntryPoint as the entry point. Note that there is a whitespace between the colon and the class name so UtilityReporterEntryPoint is seen as the main class and not as being part of the classpath.

Java code in different build environments [duplicate]

My project requires Java 1.6 for compilation and running. Now I have a requirement to make it working with Java 1.5 (from the marketing side). I want to replace method body (return type and arguments remain the same) to make it compiling with Java 1.5 without errors.
Details: I have an utility class called OS which encapsulates all OS-specific things. It has a method
public static void openFile(java.io.File file) throws java.io.IOException {
// open the file using java.awt.Desktop
...
}
to open files like with double-click (start Windows command or open Mac OS X command equivalent). Since it cannot be compiled with Java 1.5, I want to exclude it during compilation and replace by another method which calls run32dll for Windows or open for Mac OS X using Runtime.exec.
Question: How can I do that? Can annotations help here?
Note: I use ant, and I can make two java files OS4J5.java and OS4J6.java which will contain the OS class with the desired code for Java 1.5 and 1.6 and copy one of them to OS.java before compiling (or an ugly way - replace the content of OS.java conditionally depending on java version) but I don't want to do that, if there is another way.
Elaborating more: in C I could use ifdef, ifndef, in Python there is no compilation and I could check a feature using hasattr or something else, in Common Lisp I could use #+feature. Is there something similar for Java?
Found this post but it doesn't seem to be helpful.
Any help is greatly appreciated. kh.

Nope there isn't any support for conditional compilation in Java.
The usual plan is to hide the OS specific bits of your app behind an Interface and then detect the OS type at runtime and load the implementation using Class.forName(String).
In your case there no reason why you can't compile the both OS* (and infact your whole app) using Java 1.6 with -source 1.5 -target 1.5 then in a the factory method for getting hold of OS classes (which would now be an interface) detect that java.awt.Desktop
class is available and load the correct version.
Something like:
public interface OS {
void openFile(java.io.File file) throws java.io.IOException;
}
public class OSFactory {
public static OS create(){
try{
Class.forName("java.awt.Desktop");
return new OSJ6();
}catch(Exception e){
//fall back
return new OSJ5();
}
}
}

Hiding two implementation classes behind an interface like Gareth proposed is probably the best way to go.
That said, you can introduce a kind of conditional compilation using the replace task in ant build scripts. The trick is to use comments in your code which are opened/closed by a textual replacement just before compiling the source, like:
/*{{ Block visible when compiling for Java 6: IFDEF6
public static void openFile(java.io.File file) throws java.io.IOException {
// open the file using java.awt.Desktop
...
/*}} end of Java 6 code. */
/*{{ Block visible when compiling for Java 5: IFDEF5
// open the file using alternative methods
...
/*}} end of Java 5 code. */
now in ant, when you compile for Java 6, replace "IFDEF6" with "*/", giving:
/*{{ Block visible when compiling for Java 6: */
public static void openFile(java.io.File file) throws java.io.IOException {
// open the file using java.awt.Desktop
...
/*}} end of Java 6 code. */
/*{{ Block visible when compiling for Java 5, IFDEF5
public static void openFile(java.io.File file) throws java.io.IOException {
// open the file using alternative methods
...
/*}} end of Java 5 code. */
and when compiling for Java 5, replace "IFDEF5". Note that you need to be careful to use // comments inside the /*{{, /*}} blocks.

You can make the calls using reflection and compile the code with Java 5.
e.g.
Class clazz = Class.forName("java.package.ClassNotFoundInJavav5");
Method method = clazz.getMethod("methodNotFoundInJava5", Class1.class);
method.invoke(args1);
You can catch any exceptions and fall back to something which works on Java 5.

The Ant script introduced below gives nice and clean trick.
link: https://weblogs.java.net/blog/schaefa/archive/2005/01/how_to_do_condi.html
in example,
//[ifdef]
public byte[] getBytes(String parameterName)
throws SQLException {
...
}
//[enddef]
with Ant script
<filterset begintoken="//[" endtoken="]">
<filter token="ifdef" value="${ifdef.token}"/>
<filter token="enddef" value="${enddef.token}"/>
</filterset>
please go to link above for more detail.

In java 9 it's possible to create multi-release jar files. Essentially it means that you make multiple versions of the same java file.
When you compile them, you compile each version of the java file with the required jdk version. Next you need to pack them in a structure that looks like this:
+ com
+ mypackage
+ Main.class
+ Utils.class
+ META-INF
+ versions
+ 9
+ com
+ mypackage
+ Utils.class
In the example above, the main part of the code is compiled in java 8, but for java 9 there is an additional (but different) version of the Utils class.
When you run this code on the java 8 JVM it won't even check for classes in the META-INF folder. But in java 9 it will, and will find and use the more recent version of the class.

I'm not such a great Java expert, but it seems that conditional compilation in Java is supported and easy to do. Please read:
http://www.javapractices.com/topic/TopicAction.do?Id=64
Quoting the gist:
The conditional compilation practice is used to optionally remove chunks of code from the compiled version of a class. It uses the fact that compilers will ignore any unreachable branches of code.
To implement conditional compilation,
define a static final boolean value as a non-private member of some class
place code which is to be conditionally compiled in an if block which evaluates the boolean
set the value of the boolean to false to cause the compiler to ignore the if block; otherwise, keep its value as true
Of course this lets us to "compile out" chunks of code inside any method. To remove class members, methods or even entire classes (maybe leaving only a stub) you would still need a pre-processor.

if you don't want conditionally enabled code blocks in your application then a preprocessor is only way, you could take a look at java-comment-preprocessor which can be used for both maven and ant projects
p.s.
also I have made some example how to use preprocessing with Maven to build JEP-238 multi-version JAR without duplication of sources

Java Primitive Specializations Generator supports conditional compilation:
/* if Windows compilingFor */
start();
/* elif Mac compilingFor */
open();
/* endif */
This tool has Maven and Gradle plugins.

hi I have got similar problem when I have shared library between Java SDK abd Android and in both environments are used the graphics so basically my code must to work with both
java.awt.Graphics and android.graphics.Canvas,
but I don't want to duplicate almost any code.
My solution is to use wrapper, so I access to graphisc API indirectl way, and
I can change a couple of imports, to import the wrapper I want to compile the projects.
The projects have some cone shaded and some are separate, but there is no duplicating anything except of couple of wrappers etc.
I think it is the best what I can do.

Sourcing r-files only once on Rserve

I have written a small Java client which does some calculations on an Rserver. For this purpose, there are functions.r- and libraries.r files on the server side, which have to be sourced, before the actual calculation can be done.
Currently I load the files on every new connection:
import org.rosuda.REngine.Rserve.RConnection;
public class RserveTester {
public void doOnRserve() {
RConnection c = new RConnection( "rserve.domain.local" );
c.login( "foo", "user" );
c.eval("source(\"/home/rserve/lib/libraries.r\")");
c.eval("source(\"/home/rserve/lib/functions.r\")");
c.eval( "someCalculation()" )
c.close();
}
}
where doOnRserve() is called due to some events on the client side couple of times in a minute.
My Question is: Is it possibility to source the libraries only once, such that they are available during all new RSessions without individual sourcing?
I tried on the client side something like:
c.serverSource("/home/rserve/lib/libraries.r" )
c.serverSource("/home/rserve/lib/functions.r" )
Which gives me te following exception (no idea why this does not work wile eval does):
org.rosuda.REngine.Rserve.RserveException: serverSource failed, request status: access denied (local to the server)
Can I start the Rserve with a specific .Rprofile?
EDIT:
Basically, there seam to be three possible methods:
Let the /home/rserve/.Rprofile source the .r files. But this seams to source them each time I call new RConnection()
Passing the source commands directly to R when starting Rserve (no idea how to do this).
My preferred method: doing it from the client side using serverSource(), which throws these "access denied" exceptions.
EDIT2:
Rserve version v0.6-8 (338)
R version 2.15.2 for x86_64-pc-linux-gnu.

This is trivially done by adding source lines to your configuration file, i.e., putting
source "/foo/bar.R"
in /etc/Rserv.conf will source /foo/bar.R on startup. If you want to use another config file, use --RS-conf command line argument to specify it. Finally, Rserve 1.x supports --RS-source option on the command line as well.
Without the quotations in the filepath, it may give File Not Found Error.
BTW: you mentioned serverSource() access denied - that means you did not enable control commands in Rserve (control enable in the configuration or --RS-enable-control on the command line).
PS: Please use stats-rosuda-devel mailing list for Rserve questions.

Yes you can. Always remember this:
R> fortunes::fortune("Yoda")
Evelyn Hall: I would like to know how (if) I can extract some of the information
from the summary of my nlme.
Simon Blomberg: This is R. There is no if. Only how.
-- Evelyn Hall and Simon 'Yoda' Blomberg
R-help (April 2005)
R>
Or as the documentation for Rserve states:
\description{ Starts Rserve in daemon mode (unix only).
Any additional
parameters not related to Rserve will be passed straight to the
underlying R. For configuration, usage and command line parameters
please consult the online documentation at
http://www.rforge.net/Rserve. Use \code{R CMD Rserve --help} for a
brief help.

Setting the last-modified-time of a directory opened for ReadDirectoryChangesW

I hava a Java program that needs to monitor a directory tree for changes. I have JNI code that uses ReadDirectoryChangesW(). The directory is opened like:
HANDLE dirHandle = CreateFile(
path, FILE_LIST_DIRECTORY, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL,
OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL
);
and then I pass dirHandle to ReadDirectoryChangesW(). All of that works just fine.
The problem is that other parts of the code (on the Java side) use File.setLastModified() to "touch" files or directories (update their timestamps to be "now"). This generally works; however, it fails when it tried to "touch" the directory that was opened using CreateFile().
To see what Windows error is actually occurring, I looked at the JDK source for File.setLastModified() and reimplemented it in my own code with the addition of printing the error from GetLastError(); the error is:
ERROR_SHARING_VIOLATION (error 32)
"The process cannot access the file because it is being used by another process."
WTF? It's the same process. I even passed FILE_SHARE_READ and FILE_SHARE_WRITE to CreateFile().
Is there a way to make this work?
More Info
The native code implementation of File.setLastModified() in the JDK does a:
h = CreateFileW(pathbuf, GENERIC_WRITE, 0, NULL, OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL | FILE_FLAG_BACKUP_SEMANTICS, 0);
If I change the first 0 to FILE_SHARE_READ | FILE_SHARE_WRITE, it all works. So it seems that the JDK implementation is a little broken. :(
So my question now becomes: Is there a way to make this work without having to use my own (re)implementation of File.setLastModified()?

Although the error message is a bit misleading in this case, what you're seeing is normal behaviour.
By opening the directory with dwShareMode set to zero, the JDK is, in effect, asking for exclusive access, which will cause any other access attempt to fail with a sharing violation error. This applies equally to accesses
from other processes and from within your own process.
The CreateFile documentation describes the dwShareMode parameter:
If this parameter is zero and CreateFile succeeds, the file or device cannot be shared and cannot be opened again until the handle to the file or device is closed.
You cannot request a sharing mode that conflicts with the access mode that is specified in an existing request that has an open handle. CreateFile would fail and the GetLastError function would return ERROR_SHARING_VIOLATION.
So, it seems you've answered your own question: you need a custom setLastModified function that specifies FILE_SHARE_READ | FILE_SHARE_WRITE when accessing the directory.

In Java what exactly does File.canExecute() do?

I have created a plain file which does not have execute permission but when I create a Java File object using this file's path/name and then call File.canExecute() I get true as the result, whereas I would expect this method call to return false. Can someone explain what I'm missing here?
Solaris:
$ touch /tmp/nonexecutable
$ ls -l /tmp/nonexecutable
-rw-r--r-- 1 root root 0 May 21 07:48 /tmp/nonexecutable
Java:
String pathName = "/tmp/nonexecutable";
File myFile = new File(pathName);
if (!myFile.canExecute())
{
String errorMessage = "The file is not executable.";
log.error(errorMessage);
throw new RuntimeException(errorMessage);
}
Thanks in advance for your help.
--James

Nothing to do with Java - you're running as root, and root is allowed everything, not matter what the permissions say.

Though I'm not an expert, and this will not answer your question properly, I'd like to add that this behavior is not specific to Java. From the find (GNU findutils) 4.4.0 manpage on my Ubuntu 8.10 install, regarding the -executable flag:
Matches files which are
executable and directories which are
searchable (in a file name resolution
sense). This takes into account
access control lists and other
permissions artefacts which the -perm
test ignores. This test makes
use of the access(2) system call,
and so can be fooled by NFS servers
which do UID mapping (or
root-squashing), since many systems
implement access(2) in the client’s
kernel and so cannot make use of the
UID mapping information held on the
server. Because this test is
based only on the result of the
access(2) system call, there is no
guarantee that a file for which this
test succeeds can actually be
executed.

Here is a bug which was opened on JDK on this:
http://bugs.sun.com/bugdatabase/view_bug.do;jsessionid=8b833c54cb93d6c9cf416667dc02?bug_id=6379654
The conclusion is that File.canExecute() simply translates into a native posix call to access(path, X_OK). Linux returns false and solaris returns true for that call when run as root.
Finally, the bug was closed as Wont Fix! :)