We Keep Coding

How to map in JNA an array of structure inside a structure without setting the array size on the C++ declaration

I'm trying to map a structure which contains an array of structure in JNA.
The size of the embedded structure is not defined in advanced in the C++ declaration of the structure but in the java code.
My issue is that I get Exception in thread "main" java.lang.Error: Invalid memory access
The C++ header file is as follows :
typedef struct s_Param
{
char* key;
uint32_t key_value;
} Param;
typedef struct s_ParamList {
Param* init_param;
int param_list_size; // number of param items in the param List
} ParamList;
The C++ code is as follows :
int Init(
ParamList* i_initParamList,
logfunction i__callback,
const allocator* i__allocator,
void** o__content) {
...
if (i_initParamList != NULL){
printf("DLL PRINT init ---- i_initParamList = %p\n", i_initParamList);
printf("DLL PRINT init ---- i_initParamList->param_list_size = %i\n", i_initParamList->param_list_size);
if (i_initParamList->init_param == NULL){
printf("DLL PRINT init ---- i_initParamList->init_param must not be NULL\n");
returnedCode = 1;
}else{
for (int i = 0; i<i_initParamList->param_list_size;i++){
printf("DLL PRINT init ---- i_initParamList->init_param[i]->key = %s\n",i_initParamList->init_param[i].key);
printf("DLL PRINT init ---- i_initParamList->init_param[i]->key_value = %i\n",i_initParamList->init_param[i].key_value);
}
...
}
The java JNA code is as follows :
public interface MyLibrary extends Library {
#FieldOrder({ "key", "key_value" })
public static class Param extends Structure {
public static class ByReference extends Param implements Structure.ByReference {
}
public String key;
public int key_value;
public Param(){ // NOT sure that this adding constructor makes sense
super();
setAlignType(ALIGN_NONE);
}
}
#FieldOrder({ "init_param", "param_list_size" })
public static class ParamList extends Structure {
public static class ByReference extends ParamList implements Structure.ByReference {
}
public Param[] init_param;
public int param_list_size;
ParamList(){ // NOT sure that this adding constructor makes sense
super();
setAlignType(ALIGN_NONE);
}
}
public int Init(ParamList i_initParamList, logfunction i__callback, allocator i__allocator,
PointerByReference o__content);
...
}
The Sample.java code which calls the JNA library is as follows :
Note that I did not add the way the other parameters of Init function are managed as I do not have any issues with them.
int paramListSize = 4;
MyLibrary.Param[] params = new MyLibrary.Param[paramListSize];
for (int i = 0; i < paramListSize; i++) {
params[i] = new MyLibrary.Param();
}
params[0].key = "first";
params[0].key_value = 1;
params[1].key = "second";
params[1].key_value = 5;
params[2].key = "third";
params[2].key_value = 7;
params[3].key = "forth";
params[3].key_value = 9;
MyLibrary.ParamList paramList = new MyLibrary.ParamList.ByReference();
paramList.init_param = params;
paramList.param_list_size = paramListSize;
logger.debug("params = "+ params);
logger.debug("paramList = "+ paramList);
logger.debug("paramList.param_list_size = "+paramList.param_list_size);
int errInit = IFunctions.Init(paramList, logCallback, i__allocator, o__content);
The traces results coming from C++ and Java code are as follows :
10:41:28,303 DEBUG Sample:193 - params = [MyLibrary$Param;#1e67a849
10:41:28,312 DEBUG Sample:194 - paramList = MyLibrary$ParamList$ByReference(auto-allocated#0x1f3d49fe5b0 (52 bytes)) {
MyLibrary$Param init_param[4]#0x0=[MyLibrary$Param;#1e67a849
int param_list_size#0x30=0x0004
}
10:41:28,316 DEBUG Sample:195 - paramList.param_list_size = 4
Exception in thread "main" java.lang.Error: Invalid memory access
at com.sun.jna.Native.invokeInt(Native Method)
at com.sun.jna.Function.invoke(Function.java:426)
at com.sun.jna.Function.invoke(Function.java:361)
at com.sun.jna.MyLibrary$Handler.invoke(MyLibrary.java:265)
at com.sun.proxy.$Proxy3.init(Unknown Source)
at Sample.main(Sample.java:216)
DLL PRINT init ---- i_initParamList = 000001F3D49FE5B0
DLL PRINT _init ---- init_param address = 0000021B4ADAF3D0
DLL PRINT init ---- i_initParamList->param_list_size = 1
DLL PRINT init ---- i_initParamList->init_param[i]->key =
It works if I declare my Structure in C++ header as follows :
typedef struct s_ParamList {
Param init_param[4];
int param_list_size; // number of param items in the param List
} ParamList;
But I do not want to define the init_param array size in the C++ code as it must be defined on java code side.
Regarding added following code in the javacode :
ParamList(){ // NOT sure that this adding constructor makes sense
super();
setAlignType(ALIGN_NONE);
}
I'm not sure that I have to add this in both structures i.e in ParamList and in Param structure.
But anyway if I delete both of those constructor I have exactly the same issue.
I saw another way to manage my requirement (looking at toArray(size) method from chapter Structure of https://javadoc.io/doc/net.java.dev.jna/jna/latest/index.html for 5.8.0 JNA version.
Indeed I also followed the link How to fill an array of structures in JNA? but I get Invalid Memory access. I'll create a separate question regarding this second kind of implementation if I cannot have my Structure's array in a Structure.
And I had to split the Param array and the size of the array in 2 separate parameters of Init function instead of setting them into a unique Structure.
And as I already have many parameters in my Init function I would prefer to have only one structure parameter. And I guess that It should be possible in JNA.
Does any one have a clue?
Thanx for your help.

The problem is in your mapping of this structure
typedef struct s_ParamList {
Param* init_param;
int param_list_size; // number of param items in the param List
} ParamList;
The * indicates this is a pointer to memory elsewhere that you need to define.
Structures are treated as ByValue by default inside structures, and as ByRefrence by default in function arguments. So here you need to explicitly define the ByReference version of the structure (or use a plain Pointer, which is less type safe.)
So you'll get this as the main part of your structure.
#FieldOrder({ "init_param", "param_list_size" })
public static class ParamList extends Structure {
public Param.ByReference init_param;
public int param_list_size;
}
Next, you've indicated you want to define this array and allocate its memory yourself in Java. The important thing to remember here is that C treats arrays as contiguous memory, so you really only have two choices: allocate a large block yourself with Memory and set values at the offsets manually; or use Structure.toArray() designed for exactly this case: You start with an instantiated structure and then tell the toArray() method how many copies of it you need.
So your sample code would look like this:
int paramListSize = 4;
// Note the syntax for allocating a contiguous array
MyLibrary.Param.ByReference[] params =
(MyLibrary.Param.ByReference[]) new MyLibrary.Param.ByReference().toArray(paramListSize);
// set the values as you've alread done
params[0].key = "first";
params[0].key_value = 1;
// and so on...
// Now instantiate your structure and set its members
MyLibrary.ParamList paramList = new MyLibrary.ParamList();
// The first array member is the pointer to the start of the array
paramList.init_param = params[0];
paramList.param_list_size = paramListSize;
And here, you pass it to the native function. It is ByReference by default.
You can find another general example here

Can you consider following data structure?
typedef struct s_ParamList {
int param_list_size; // number of param items in the param List
Param init_param[0];
} ParamList;
If yes, please be accurate with memory allocation and array usage, since such approach is pretty dangerous and may easily cause data overrun

Unable to set Java int array field from JNI

I'm developing an Android App and I'm receiving camera data from a lib in C++. I need to send this data from C++ to the Java code. For this, I'm using JNI. I'm able to set different fields in Java from the JNI and the C++ data (like the name or the type of the camera), but I'm unable to set the ID Field because it's an uint8_t array.
How can I do this?
I already tried several ways to do this but each time I've got an SIGSEGV error with an invalid address. For others fields I'm using
env->Set<Primitives>Field(jobject, jfieldID, value)
method but there are no methods like that for int array, are there?
So, I've tried to set this field by calling a method from my class and provide the int array as parameter but this function failed and returned the SIGSEGV error.
Then, I searched on the web and I tried to set the field through
env->GetObjectField(jobject, jfieldID)
and
env->SetIntArrayRegion(jintArray, start, end, myIntArray)
but here the first method returns always null.
JavaVM * mJVM; //My Java Virtual Machine
jobject mCameraObject, mThreadObject; //Previously initialize to call functions in the right thread
void onReceiveCameraList(void *ptr, uint32_t /*id*/, my::lib::Camera *arrayCamera, uint32_t nbCameras) {
JNIEnv *env;
mJVM->AttachCurrentThread(&env, nullptr);
if (env->ExceptionCheck())
return;
//Get Field, Method ID, Object and Class
jclass cameraClass = env->GetObjectClass(mCameraObject);
jfieldID camIDField = env->GetFieldID(cameraClass, "idCam", "[I");
jfieldID camNameField = env->GetFieldID(cameraClass, "label", "Ljava/lang/String;");
jfieldID camConnectedField = env->GetFieldID(cameraClass, "connected", "Z");
jfieldID camTypeField = env->GetFieldID(cameraClass, "typeProduit", "B");
jmethodID camReceptionMID = env->GetMethodID(env->GetObjectClass(mThreadObject), "onCamerasReception", "([Lcom/my/path/models/Camera;)V"); //Java function
jobjectArray cameraArray = env->NewObjectArray(nbCameras, cameraClass, mCameraObject); //Object return in the functions
//Put the cameras into the vector
std::vector<my::lib::Camera> vectorCameras;
if(!vectorCameras.empty())
vectorCameras.clear();
if ((arrayCamera != nullptr) && (nbCameras > 0)) {
for (uint32_t i = 0; i < nbCameras; ++i) {
vectorCameras.push_back(arrayCamera[i]);
}
}
//Set the my::lib::Camera field into Java::Camera
int c= 0;
for (auto & cam : vectorCameras)
{
jobject camera = env->AllocObject(cameraClass); //Object Camera to add in cameraArray object
// MY DATA TO SET ID FIELD ///
jint idArray[16];
for (int i = 0; i < 16 ; ++i) {
idArray[i] = cam.idCamera.data[i]; // uint8_t cam.idCamera.data[16]
}
///////// FIRST WAY /////////
jmethodID setIDCamMID = env->GetMethodID(env->GetObjectClass(camera), "setIDCam", "([I)V");
env->CallVoidMethod(camera, setIDCamMID, idArray);
///////// SECOND WAY /////////
jintArray jintArray1 = (jintArray)env->GetObjectField(camera, camIDField);
env->SetIntArrayRegion(jintArray1, 0, 16, idArray);
//Set<Primitives>Field : WORKING
env->SetObjectField(camera, camNameField, env->NewStringUTF((const char *) cam.labelCamera));
env->SetBooleanField(camera, camConnectedField, cam.isCameraConnected);
jbyte type;
if (cam.typeCamera == my::lib::TYPE_1 || cam.typeCamera == my::lib::TYPE_2 || cam.typeCamera == my::lib::TYPE_3) //type not known in JAVA
type = 0;
else
type = cam.typeCamera;
env->SetByteField(camera, camTypeField, type);
//Put camera object into cameraArray object
env->SetObjectArrayElement(cameraArray, c++, camera);
}//for
//Call Java method with cameraArray
env->CallVoidMethod(mThreadObject, camReceptionMID, dpCameraArray);
}//onreceiveCamera
Can someone tell me if I made a mistake or am using it the wrong way?
Is there any other way to set this data?

This yields a C++ array with elements of type jint:
// MY DATA TO SET ID FIELD ///
jint idArray[16];
for (int i = 0; i < 16 ; ++i) {
idArray[i] = cam.idCamera.data[i]; // uint8_t cam.idCamera.data[16]
}
It is important to understand that that is not a Java array. Therefore, this ...
///////// FIRST WAY /////////
jmethodID setIDCamMID = env->GetMethodID(env->GetObjectClass(camera), "setIDCam", "([I)V");
env->CallVoidMethod(camera, setIDCamMID, idArray);
... is incorrect. idArray is not the right type (and does not decay to a pointer to the right type) for the corresponding parameter to the Java method you are trying to invoke.
On the other hand, this ...
///////// SECOND WAY /////////
jintArray jintArray1 = (jintArray)env->GetObjectField(camera, camIDField);
env->SetIntArrayRegion(jintArray1, 0, 16, idArray);
... is ok, provided that the field already contains a reference to an int[] of length at least 16. Since it didn't work for you, I take it that the field's initial value does not satisfy that criterion.
If you need to create a new Java int[], then
use JNI's NewIntArray() function, which returns a jintArray with the length you specify. Then
use the appropriate JNI methods for setting the elements (see below), and finally
either assign the array directly to the target object's field with SetObjectField() or use the object's setter method to do so, as in your first attempt.
As for setting elements of the Java array, SetIntArrayRegion() will work fine for that (given an actual Java array of sufficient length), but that does require you to allocate a separate native array of primitives (your idArray) from which to copy the values. A slightly more efficient approach would be to use GetPrimitiveArrayCritical() to let Java provide the buffer -- possibly a direct pointer to the internal data -- and then ReleasePrimitiveArrayCritical() when you're done. Something like this:
// It is assumed here that the length of the array is sufficient, perhaps because
// we just created this (Java) array.
jint *idArray = (jint *) env->GetPrimitiveArrayCritical(jintArray1, NULL);
for (uint32_t i = 0; i < nbCameras; ++i) {
idArray[i] = cam.idCamera.data[i];
}
env->ReleasePrimitiveArrayCritical(jintArray1, idArray, 0);

For the first approach, you need to create a Java int[] first, fill it from idArray, and then call your method:
jintArray j_arr = env->NewIntArray(16);
env->SetIntArrayRegion(j_arr, 0, 16, idArray);
env->CallVoidMethod(camera, setIDCamMID, j_arr);
Your second approach does not work because you never called a constructor that filled the idCam field.
You can do that from JNI, however:
env->SetObjectField(camera, camIDField, j_arr);

How to copy native memory to DirectByteBuffer

I know one way - using memcpy on C++ side:
C++ method:
void CopyData(void* buffer, int size)
{
memcpy(buffer, source, size);
}
JNR mapping:
void CopyData(#Pinned #Out ByteBuffer byteBuffer, #Pinned #In int size);
Java invocation:
ByteBuffer buffer = ByteBuffer.allocateDirect(size);
adapter.CopyData(buffer, size);
But I would like to handle case when native code does not copy data, but only returns pointer to the memory which is to be copied:
C++ methods:
void* GetData1()
{
return source;
}
// or
struct Data
{
void* data;
};
void* GetData2(Data* outData)
{
outData->data = source;
}
I know how to write JNR mapping to be able to copy data to HeapByteBuffer:
Pointer GetData1();
// or
void GetData2(#Pinned #Out Data outData);
final class Data extends Struct {
public final Struct.Pointer data;
public DecodeResult(Runtime runtime) {
super(runtime);
data = new Struct.Pointer();
}
}
Java invocation:
ByteBuffer buffer = ByteBuffer.allocate(size);
Pointer dataPtr = adapter.GetData1();
dataPtr.get(0, buffer.array(), 0, buffer.array().length);
// or
ByteBuffer buffer = ByteBuffer.allocate(size);
Data outData = new Data(runtime);
adapter.GetData2(outData);
Pointer dataPtr = outData.data.get();
dataPtr.get(0, buffer.array(), 0, buffer.array().length);
But I have not found a way to copy memory to DirectByteBuffer instead of HeapByteBuffer. The above snippet of code does not work for DirectByteBuffer because buffer.array() is null for such a buffer, as it is backed by native memory area.
Please help.

I have found several ways to perform copying of JNR native memory to DirectByteBuffer. They differ in efficiency. Currently I use the following aproach, I don't know whether is it the best or intended by JNR authors:
ByteBuffer buffer = ByteBuffer.allocateDirect(size);
Pointer dataPtr = adapter.GetData1();
long destAddress = ((DirectBuffer)buffer).address();
Pointer destPtr = AsmRuntime.pointerValue(destAddress, runtime);
assert dataPtr.isDirect() && destPtr.isDirect();
dataPtr.transferTo(0, destPtr, 0, size);
or
ByteBuffer buffer = ByteBuffer.allocateDirect(size);
Data outData = new Data(runtime);
adapter.GetData2(outData);
Pointer dataPtr = outData.data.get();
long destAddress = ((DirectBuffer)buffer).address();
Pointer destPtr = AsmRuntime.pointerValue(destAddress, runtime);
assert dataPtr.isDirect() && destPtr.isDirect();
dataPtr.transferTo(0, destPtr, 0, size);
It is important that the assert clause above is fulfilled. It guarantees that pointers are jnr.ffi.provider.jffi.DirectMemoryIO instances, and the efficient memcpy method is used for copying (check implementation of DirectMemoryIO.transferTo()).
The alternative is to wrap DirectByteBuffer using the following method:
Pointer destPtr = Pointer.wrap(runtime, destAddress);
or
Pointer destPtr = Pointer.wrap(runtime, destAddress, size);
but no:
Pointer destPtr = Pointer.wrap(runtime, buffer);
The first and second pointers are backed by DirectMemoryIO, but the third pointer is backed by ByteBufferMemoryIO and it involves slow byte-by-byte copying.
The one drawback is that DirectMemoryIO instance is quite heavyweight. It allocates 32 bytes on JVM heap, so in case of plenty of JNR invocations, all DirectMemoryIO instances consume big part of memory.

IntByReference returns 0 instead of the actual value

I have the following C code which I am calling from JNA. When I call it I am getting 0 as the return value while I'm getting the real value when I test it in C.
RFIDLIB_API uint32_t get(
ipj_key key /*[i]*/,
uint32_t bank_index /*[in]*/,
uint32_t value_index /*[out]*/,
uint32_t *value /*[out]*/)
{
return ipj_get(&iri_device,key,bank_index,value_index,value);
}
Below is how I have defined this in my JNA library.
public int get(ipj_key key, int bank_index, int value_index, IntByReference value);
Below is the ip_key structure
public class ipj_key extends Structure {
public int ipj_key;
#Override
protected List getFieldOrder() {
return Arrays.asList("ipj_key");
}
}
And below is how I am calling it in my main method.
public rfidlib rlib;
public static void main(String[] args) {
MainScannerTest mainClass = new MainScannerTest();
mainClass.rlib = (rfidlib) Native.loadLibrary("rfidlib", rfidlib.class);
ipj_key key = new ipj_key();
key.ipj_key = 0xD;
IntByReference value = new IntByReference();
mainClass.rlib.get(key, 0, 0, value);
System.out.println("Antenna power is : "+ value.getValue());
}
What am I doing wrong here? Why am I getting 0 as the return value? Please advice.

My C lib has a int* as an OUT parameter. I was also trying to use it with an IntByReference, without any success.
EDIT 2017-03-16: indeed it worked with IntByReference. There might me something wrong in your C code.
An alternative would be using a Pointer instead of IntByreference
(I made it work with IntByRef and also with Pointer in my case (64bit Win7, JNA 4.3.0 and Java 64 bits jdk 1.8.0_71))
With your code snippet, this would look like the following.
First change your JNA dll interface defintion (use Pointer instead of IntByReference):
public int get(ipj_key key, int bank_index, int value_index, /*change is here*/ Pointer value);
Then change of course the variable before call:
Pointer value = new Memory(Integer.size); // Here we assume that your java platform is 32 bits (Integer.size=32)
//this should match your uint32_t. No idea if there are consequences when mixing 32/64 bits java/OS/dll ...
Then, you can do the following:
System.out.println("Antenna power is : "+ value.getInt(0));
// The offset is set to 0, since there is no reason to read the int from another offset value.

How to pass an array of structures using JNA

I referenced another stackoverflow article "How do I map a pointer to an array of structures in JNA" to come up with the following code to enumerate Windows service dependencies.
Structure and function declarations:
static class SERVICE_STATUS extends Structure {
public int dwServiceType;
public int dwCurrentState;
public int dwControlsAccepted;
public int dwWin32ExitCode;
public int dwServiceSpecificExitCode;
public int dwCheckPoint;
public int dwWaitHint;
public SERVICE_STATUS(){}
}
static class ENUM_SERVICE_STATUS extends Structure {
public ENUM_SERVICE_STATUS(){ }
public WString lpServiceName;
public WString lpDisplayName;
SERVICE_STATUS serviceStatus;
}
boolean EnumDependentServicesW(Pointer hService, int serviceState, ENUM_SERVICE_STATUS serviceStatuses, int size, IntByReference bytesNeeded, IntByReference servicesReturned);
If there is only one service dependency, the following code works:
IntByReference bytesNeeded = new IntByReference();
IntByReference numberOfServices = new IntByReference();
Advapi32.ENUM_SERVICE_STATUS serviceStatus = new Advapi32.ENUM_SERVICE_STATUS();
Advapi32.ENUM_SERVICE_STATUS[] serviceStatuses = (Advapi32.ENUM_SERVICE_STATUS[]) serviceStatus.toArray(1);
if (!advapi32.EnumDependentServicesW(serviceHandle, Advapi32.SERVICE_ACTIVE, null, 0, bytesNeeded, numberOfServices)) {
if (advapi32.EnumDependentServicesW (serviceHandle, Advapi32.SERVICE_ACTIVE, serviceStatuses[0], bytesNeeded.getValue(), bytesNeeded, numberOfServices)) {
for(int i = numberOfServices.getValue() - 1; i >= 0; i--){
logger.debug("Service Name: " + serviceStatuses[i].lpServiceName.toString());
}
}
If there are 2 service dependencies, I get a NullPointerException for lpServiceName in the logger.debug call:
IntByReference bytesNeeded = new IntByReference();
IntByReference numberOfServices = new IntByReference();
Advapi32.ENUM_SERVICE_STATUS serviceStatus = new Advapi32.ENUM_SERVICE_STATUS();
Advapi32.ENUM_SERVICE_STATUS[] serviceStatuses = (Advapi32.ENUM_SERVICE_STATUS[]) serviceStatus.toArray(2);
if (!advapi32.EnumDependentServicesW(serviceHandle, Advapi32.SERVICE_ACTIVE, null, 0, bytesNeeded, numberOfServices)) {
if (advapi32.EnumDependentServicesW (serviceHandle, Advapi32.SERVICE_ACTIVE, serviceStatuses[0], bytesNeeded.getValue(), bytesNeeded, numberOfServices)) {
for(int i = numberOfServices.getValue() - 1; i >= 0; i--){
logger.debug("Service Name: " + serviceStatuses[i].lpServiceName.toString());
}
}
The numberOfServices value for the code above is 2, as expected. I'm trying to pass the structure array instead of a pointer because I want JNA to do the memory synching. How should I be passing/using the array of structures?

According to the docs for EnumDependentServices,
lpServices [out, optional]
A pointer to an array of
ENUM_SERVICE_STATUS structures that receives the name and service
status information for each dependent service in the database. The
buffer must be large enough to hold the structures, plus the strings
to which their members point.
You're pretty much ignoring the required buffer size as reported by bytesNeeded. You should use the bytesNeeded value to create a Memory instance of the requisite size, then use that Memory instance to create a new ENUM_SERVICE_STATUS instance, instead of creating the structure independently of the required buffer size.