We Keep Coding

Using rsync4j library, a Java implementation of Linux rsync tool, how to sync data from a remote directory with a local directory

I need the functionality like that of the rsync linux tool in my Java program. For that, I chose the rsync4j library.
Using their documentation, I wrote the following program:
import com.github.fracpete.processoutput4j.output.ConsoleOutputProcessOutput;
import com.github.fracpete.rsync4j.RSync;
public class MainClass {
public static void main(String [] args) {
System.out.println("Started");//check
RSync rsync = new RSync()
.source("/home/arth/DataSourceFolder/a.txt")
.destination("/home/arth/DataDestinationFolder/")
.recursive(true);
// or if you prefer using commandline options:
// rsync.setOptions(new String[]{"-r", "/one/place/", "/other/place/"});
CollectingProcessOutput output = null;
try {
System.out.println("Inside try");
output = rsync.execute();
System.out.println("End of try");
} catch (Exception e) {
e.printStackTrace();
}
System.out.println(output.getStdOut());
System.out.println("Exit code: " + output.getExitCode());
if (output.getExitCode() > 0)
System.err.println(output.getStdErr());
}
}
In the snippet, in out local machine, a file a.txt is copied from one location to another. This works perfectly. The file is successfully copied when I run it and here is the output:
Started
Inside try
End of try
Exit code: 0
But my need is to sync a local directory with a directory lying at a remote host/machine. When I tried to do it using a simple rsync command from a terminal using the following command
rsync remoteUserName#23.24.25.244:/home/beth/remoteFolder/a.png /home/arth/DataSourceFolder
it works like a charm. a.png IS copied to local machine at path specified, although a password of remote machine is asked first.
But the problem when I use the above Java program to do the same operation, by replacing line # 11 and 12 by:
.source("remoteUserName#23.24.25.244:/home/beth/remoteFolder/a.png")
.destination("/home/arth/DataDestinationFolder/")
the program gets stuck after printing Started in the console. Neither an exception is thrown nor does the program proceed.
The question is that how do I fix this problem?

(old post, I know, but here it goes...) The rsync4j library does not allow interaction. In your case, the underlying rysnc binary prompts for a password in the process that the Java library created, but never receives one.
Starting with release 3.2.3-7, you can supply an instance of the sshpass wrapper to feed in the password (see this comment for an example).

Getting an UnsatisfiedLinkError (undefined symbol) in Java while loading JNI dependencies even after successfully loading the required object file

I'm using Google OR-tools library (v6.4) for a project (though my question is not specific to this library). This consists of one jar, which has a few native dependencies (a bunch of ".so"/".dylib" object files, depending on the OS). This build for my project is being made on Ubuntu 14.04
The problem I'm facing: On trying to load a specific object file at runtime (using System.load()), I'm getting an UnsatisfiedLinkError with the message as "undefined symbol" (I've added the stacktrace below). However, I am loading the object file defining this symbol just before this, so I'm not sure why this error is being thrown.
I'm loading the dependencies in the following way: The object files are being packed into the jar created by Maven during build, and are being extracted and loaded (using System.load()) at runtime. The method for that is as follows:
public class EnvironmentUtils {
public static void loadResourceFromJar(String prefix, String suffix) {
String tempFilesDirectory = System.getProperty("java.io.tmpdir");
File tempFile = null;
try {
tempFile = new File(tempFilesDirectory + "/" + prefix + suffix);
tempFile.deleteOnExit();
try (final InputStream inputStream = EnvironmentUtils.class.getClassLoader().
getResourceAsStream(prefix+suffix)) {
if (inputStream == null) {
throw new RuntimeException(prefix + suffix + " was not found inside JAR.");
} else {
Files.copy(inputStream, tempFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
}
}
System.load(tempFile.getAbsolutePath());
} catch (Exception e) {
//Log top 10 lines of stack trace
}
}
}
This method is being called inside a static block for all dependencies:
public class DummyClass {
static {
String sharedLibraryExtension = EnvironmentUtils.getSharedLibraryExtension(); //.so for linux, .dylib for Mac
String jniLibraryExtension = EnvironmentUtils.getJniLibraryExtension(); //.so for linux, .jnilib for Mac
EnvironmentUtils.loadResourceFromJar("libfap", sharedLibraryExtension);
EnvironmentUtils.loadResourceFromJar("libcvrptw_lib", sharedLibraryExtension);
EnvironmentUtils.loadResourceFromJar("libortools", sharedLibraryExtension);
EnvironmentUtils.loadResourceFromJar("libdimacs", sharedLibraryExtension);
EnvironmentUtils.loadResourceFromJar("libjniortools", jniLibraryExtension);
}
}
On running System.load() for libdimacs.so, an UnsatisfiedLinkError is thrown. Stacktrace:
java.lang.UnsatisfiedLinkError: /tmp/libdimacs.so: /tmp/libdimacs.so: undefined symbol: _ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at com.(PROJECT_NAME).utils.EnvironmentUtils.loadResourceFromJar(EnvironmentUtils.java:78)
at com.(PROJECT_NAME).DummyClass.<clinit>(DummyClass.java:28)
However, this symbol "_ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_" is present in libortools.so, which is being loaded before libdimacs. I verified this by running the following command:
objdump -t (LIBRARY_PATH)/libortools.so | grep _ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_
This gave me the following output:
0000000000ce12cc gw F .text 00000091 _ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_
So it would seem that the symbol should have been defined at the time of the System.load() call, unless there was some issue in loading the containing object file. To check if the object file had been loaded correctly, I used the approach detailed in this solution. Apart from the class detailed in that answer, I added the following lines after System.load() call in EnvironmentUtils.loadResourceFromJar() to print the most recently loaded library name:
public class EnvironmentUtils {
public static void loadResourceFromJar(String prefix, String suffix) {
...
System.load(tempFile.getAbsolutePath());
final String[] libraries = ClassScope.getLoadedLibraries(ClassLoader.getSystemClassLoader());
System.out.println(libraries[libraries.length - 1]);
}
}
The output (till just before the UnsatisfiedLinkError) is as follows:
/tmp/libfap.so
/tmp/libcvrptw_lib.so
/tmp/libortools.so
So libortools.so seems to be loading correctly, which means the symbol should be loaded in memory. The exact same code is working perfectly with the corresponding Mac (".dylib") dependencies (Built on MacOS Sierra 10.12.5). Would appreciate any advice on resolving this. Thank you.

I'm apologize that the java artifact may be broken currently...
you can use c++filt to demangle the symbol ;)
c++filt _ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_
google::FlagRegisterer::FlagRegisterer<bool>(char const*, char const*, char const*, bool*, bool*)
In fact gflag has recently change its namespace from google:: to gflags:: and glog or protobobuf? try to find the correct one and I guess it failed...
note: Still not completely sure whose is the bad guy who use the google:: namespace since libortools merge all its static dependencies but I guess now you understand the bug...
note2: I have a patch in mizux/shared branch https://github.com/google/or-tools/commit/805bc0600f4b5645114da704a0eb04a0b1058e28#diff-e8590fe6fb5044985c8bf8c9e73c0d88R114
warning: this branch is currently broken and not ready yet. I'm trying ,for unix, to move from static to dynamic dependencies, so I need to fix all rpath, transitives deps etc... and in the process I also had to fix this issue (that I didn't reproduced while using static dependencies)
If too long to finish (we should create a release 6.7.2 or 6.8 (i.e. new artifact) by the end of May 2018) which maybe only contains this fix and not my branch...

Could not load JIntellitype.dll from local file system or from inside JAR

I am trying to use JIntellitype to listen to global hotkeys but I get this error:
Exception in thread "main"
com.melloware.jintellitype.JIntellitypeException: Could not load
JIntellitype.dll from local file system or from inside JAR at
com.melloware.jintellitype.JIntellitype.(JIntellitype.java:114)
at
com.melloware.jintellitype.JIntellitype.getInstance(JIntellitype.java:177)
at utils.HotKey.(HotKey.java:19) at
ui.Main.Catch_Hotkeys(Main.java:78) at ui.Main.(Main.java:20)
at ui.Main.main(Main.java:15) Caused by: java.io.IOException:
FromJarToFileSystem could not load DLL:
com/melloware/jintellitype/JIntellitype.dll at
com.melloware.jintellitype.JIntellitype.fromJarToFs(JIntellitype.java:150)
at
com.melloware.jintellitype.JIntellitype.(JIntellitype.java:105)
... 5 more Caused by: java.lang.NullPointerException at
com.melloware.jintellitype.JIntellitype.fromJarToFs(JIntellitype.java:146)
... 6 more
I have loaded the jar file and I also pointed to the folder where the dlls are located through Referenced Libraries.
Here is the code I am trying to run:
import com.melloware.jintellitype.HotkeyListener;
import com.melloware.jintellitype.IntellitypeListener;
import com.melloware.jintellitype.JIntellitype;
public class HotKey extends Thread implements HotkeyListener, IntellitypeListener {
private final int CTRL_C_SHIFT = 10;
public HotKey()
{
JIntellitype.getInstance().unregisterHotKey(CTRL_C_SHIFT);
JIntellitype.getInstance().registerHotKey(CTRL_C_SHIFT, JIntellitype.MOD_CONTROL + (int)'C', JIntellitype.MOD_SHIFT);
if (!JIntellitype.isJIntellitypeSupported())
{
System.exit(1);
}
}
#Override
public void onIntellitype(int arg0)
{
}
#Override
public void onHotKey(int key)
{
if (key == CTRL_C_SHIFT)
{
System.out.println("smg");
}
}
}
Any idea how to fix this?

Your problem will occur because of a version problem between that OS version and the JRE version.
You should check:
Whether an appropriate dll file is installed in your OS system folder.
JIntellitype package has two dll files, one is for 32bit OSs and the other is for 64bit OSs, they have different names.
Check your Java Platform version in the properties of the projects.
You can try to change the Java Platform, if there are more than one types of JDKs.
Make sure about which one is for 64bit or 32bit version.
Have good luck!

I recommend you do something like this:
try
{
JIntellitype.getInstance().unregisterHotKey(CTRL_C_SHIFT);
MyHotKeyListener hotKeyListener = new MyHotKeyListener();
hotKeyListener.addObserver(new MyEventListener());
JIntellitype.getInstance().addHotKeyListener(hotKeyListener);
JIntellitype.getInstance().registerHotKey(CTRL_C_SHIFT, JIntellitype.MOD_CONTROL + (int)'C', JIntellitype.MOD_SHIFT);
}
catch (JIntellitypeException je)
{
logger.warn("JIntellitype initialization failed.");
// DO WHATEVER (NOTIFY USERS?)
}
I can point to other threads, including one where the creator of this library himself denies problems with the library. However, many users such as myself encounter these sort of problems from time to time where JIntellitype fails to initialize and the only solution is to reboot the computer. Because of this, you should catch the JIntellitype exception (the only exception thrown by the library) and warn users (via dialog window) that the hotkey failed to register. You should give them the option to continue without them, or to reboot the computer and trying again.
Trust me.... unless this is a constant problem (which means you configured it incorrectly), it is your best alternative. This WILL happen from time to time at random.

Can I simulate a specific filesystem while unit testing?

I have been googling a little and did not find an answer which suited my specific case.
I am working on a project file manager classes, and discovered that it was developped to behave differently on Windows and Unix filesystems.
More specifically, it is compensating for the case-senstivity in Unix: when a file is not found, the manager will look for it in a case-insensitive way.
Before changing this code, I would like to implement some unit tests. However, our development machines and our CIP are both on Windows, and I have no Unix machine available. The machines and IDEs are provided by the customer. Virtualization is not an option, and dual-booting is even less.
Is there a way to test both Windows and Unix mode while being platform-independant for the build? I think the ideal would be to run the whole Test Class in a mode, and then in the other, but even a more hands-on solution would be great.
In production mode, the file managers are initialized using Spring, but they are the lowest level of the chain, using directly java.io.
Versions: Java 6, JUnit 4.9

You can use Jimfs with dependency
<dependency>
<groupId>com.google.jimfs</groupId>
<artifactId>jimfs</artifactId>
<version>1.1</version>
</dependency>
Then you could create a linux, windows and Mac file systems using
FileSystem fileSystem = Jimfs.newFileSystem(Configuration.osX());
FileSystem fileSystem = Jimfs.newFileSystem(Configuration.windows());
FileSystem fileSystem = Jimfs.newFileSystem(Configuration.unix());
example
class FilePathReader {
String getSystemPath(Path path) {
try {
return path
.toRealPath()
.toString();
} catch (IOException ex) {
throw new UncheckedIOException(ex);
}
}
}
class FilePathReaderUnitTest {
private static String DIRECTORY_NAME = "baeldung";
private FilePathReader filePathReader = new FilePathReader();
#Test
#DisplayName("Should get path on windows")
void givenWindowsSystem_shouldGetPath_thenReturnWindowsPath() throws Exception {
FileSystem fileSystem = Jimfs.newFileSystem(Configuration.windows());
Path path = getPathToFile(fileSystem);
String stringPath = filePathReader.getSystemPath(path);
assertEquals("C:\\work\\" + DIRECTORY_NAME, stringPath);
}
#Test
#DisplayName("Should get path on unix")
void givenUnixSystem_shouldGetPath_thenReturnUnixPath() throws Exception {
FileSystem fileSystem = Jimfs.newFileSystem(Configuration.unix());
Path path = getPathToFile(fileSystem);
String stringPath = filePathReader.getSystemPath(path);
assertEquals("/work/" + DIRECTORY_NAME, stringPath);
}
private Path getPathToFile(FileSystem fileSystem) throws Exception {
Path path = fileSystem.getPath(DIRECTORY_NAME);
Files.createDirectory(path);
return path;
}
}
All this copied from Baeldung.

You could dualboot Ubuntu easily by installing it with wubi.
I've learnt that unit-test should not access the file system for different reasons (speed being one of them).
For Java 6 look into theese:
http://docs.oracle.com/javase/6/docs/api/javax/tools/JavaFileManager.html
http://docs.oracle.com/javase/6/docs/api/javax/swing/filechooser/FileSystemView.html
If you were to use Java 7 this might help you:
http://download.oracle.com/javase/7/docs/technotes/guides/io/fsp/filesystemprovider.html

you could use a VM to test it on unix. Virtual Box by Oracle is agood virtualization software. Install Ubuntu, or Fedora, or some other unix based OS' Disk Image. Transfer your files to the VM. You can directly check out from your source control into the VM and you should be good to go. Atleast I am assuming that is what you want to do : test your software in both windows and linux, but currently don't have linux at your disposal

Java: CaptureDeviceManager#getDeviceList() is empty?

I am trying to print out all of the capture devices that are supported using the #getDeviceList() method in the CaptureDeviceManager class and the returned Vector has a size of 0.
Why is that? I have a webcam that works - so there should be at least one. I am running Mac OS X Lion - using JMF 2.1.1e.
Thanks!

CaptureDeviceManager.getDeviceList(Format format) does not detect devices. Instead it reads from the JMF registry which is the jmf.properties file. It searches for the jmf.properties file in the classpath.
If your JMF install has succeeded, then the classpath would have been configured to include all the relevant JMF jars and directories. The JMF install comes with a jmf.properties file included in the 'lib' folder under the JMF installation directory. This means the jmf.properties would be located by JMStudio and you would usually see the JMStudio application executing correctly. (If your JMF install is under 'C:\Program Files', then run as administrator to get around UAC)
When you create your own application to detect the devices, the problem you described above might occur. I have seen a few questions related to the same problem. This is because your application's classpath might be different and might not include the environment classpath. Check out your IDE's properties here. The problem is that CaptureDeviceManager cannot find the jmf.properties file because it is not there.
As you have found out correctly, you can copy the jmf.properties file from the JMF installation folder. It would contain the correct device list since JMF detects it during the install (Check it out just to make sure anyway).
If you want do device detection yourself, then create an empty jmf.properties file and put it somewhere in your classpath (it might throw a java.io.EOFException initially during execution but that's properly handled by the JMF classes). Then use the following code for detecting webcams...
import javax.media.*;
import java.util.*;
public static void main(String[] args) {
VFWAuto vfwObj = new VFWAuto();
Vector devices = CaptureDeviceManager.getDeviceList(null);
Enumeration deviceEnum = devices.elements();
System.out.println("Device count : " + devices.size());
while (deviceEnum.hasMoreElements()) {
CaptureDeviceInfo cdi = (CaptureDeviceInfo) deviceEnum.nextElement();
System.out.println("Device : " + cdi.getName());
}
}
The code for the VFWAuto class is given below. This is part of the JMStudio source code. You can get a good idea on how the devices are detected and recorded in the registry. Put both classes in the same package when you test. Disregard the main method in the VFWAuto class.
import com.sun.media.protocol.vfw.VFWCapture;
import java.util.*;
import javax.media.*;
public class VFWAuto {
public VFWAuto() {
Vector devices = (Vector) CaptureDeviceManager.getDeviceList(null).clone();
Enumeration enum = devices.elements();
while (enum.hasMoreElements()) {
CaptureDeviceInfo cdi = (CaptureDeviceInfo) enum.nextElement();
String name = cdi.getName();
if (name.startsWith("vfw:"))
CaptureDeviceManager.removeDevice(cdi);
}
int nDevices = 0;
for (int i = 0; i < 10; i++) {
String name = VFWCapture.capGetDriverDescriptionName(i);
if (name != null && name.length() > 1) {
System.err.println("Found device " + name);
System.err.println("Querying device. Please wait...");
com.sun.media.protocol.vfw.VFWSourceStream.autoDetect(i);
nDevices++;
}
}
}
public static void main(String [] args) {
VFWAuto a = new VFWAuto();
System.exit(0);
}
}
Assuming you are on a Windows platform and you have a working web-cam, then this code should detect the device and populate the jmf.properties file. On the next run you can also comment out the VFWAuto section and it's object references and you can see that CaptureDeviceManager reads from the jmf.properties file.
The VFWAuto class is part of jmf.jar. You can also see the DirectSoundAuto and JavaSoundAuto classes for detecting audio devices in the JMStudio sample source code. Try it out the same way as you did for VFWAuto.
My configuration was Windows 7 64 bit + JMF 2.1.1e windows performance pack + a web-cam.

I had the same issue and I solved by invoking flush() on my ObjectInputStream object.
According to the API documentation for ObjectInputStream's constructor:
The stream header containing the magic number and version number are read from the stream and verified. This method will block until the corresponding ObjectOutputStream has written and flushed the header.
This is a very important point to be aware of when trying to send objects in both directions over a socket because opening the streams in the wrong order will cause deadlock.
Consider for example what would happen if both client and server tried to construct an ObjectInputStream from a socket's input stream, prior to either constructing the corresponding ObjectOutputStream. The ObjectInputStream constructor on the client would block, waiting for the magic number and version number to arrive over the connection, while at the same time the ObjectInputStream constructor on the server side would also block for the same reason. Hence, deadlock.
Because of this, you should always make it a practice in your code to open the ObjectOutputStream and flush it first, before you open the ObjectInputStream. The ObjectOutputStream constructor will not block, and invoking flush() will force the magic number and version number to travel over the wire. If you follow this practice in both your client and server, you shouldn't have a problem with deadlock.
Credit goes to Tim Rohaly and his explanation here.

Before calling CaptureDeviceManager.getDeviceList(), the available devices must be loaded into the memory first.
You can do it manually by running JMFRegistry after installing JMF.
or do it programmatically with the help of the extension library FMJ (Free Media in Java). Here is the code:
import java.lang.reflect.Field;
import java.util.Vector;
import javax.media.*;
import javax.media.format.RGBFormat;
import net.sf.fmj.media.cdp.GlobalCaptureDevicePlugger;
public class FMJSandbox {
static {
System.setProperty("java.library.path", "D:/fmj-sf/native/win32-x86/");
try {
final Field sysPathsField = ClassLoader.class.getDeclaredField("sys_paths");
sysPathsField.setAccessible(true);
sysPathsField.set(null, null);
} catch (Exception e) {
e.printStackTrace();
}
}
public static void main(String args[]) {
GlobalCaptureDevicePlugger.addCaptureDevices();
Vector deviceInfo = CaptureDeviceManager.getDeviceList(new RGBFormat());
System.out.println(deviceInfo.size());
for (Object obj : deviceInfo ) {
System.out.println(obj);
}
}
}
Here is the output:
USB2.0 Camera : civil:\\?\usb#vid_5986&pid_02d3&mi_00#7&584a19f&0&0000#{65e8773d-8f56-11d0-a3b9-00a0c9223196}\global
RGB, -1-bit, Masks=-1:-1:-1, PixelStride=-1, LineStride=-1