We Keep Coding

Error while trying to write on parquet file in datastage 11.7 (File_Connector_20,0: java.lang.NoClassDefFoundError: org.apache.hadoop.fs.FileSystem)

we have recently upgraded the DataStage from 9.1 to 11.7 on Server AIX 7.1 .
and i'm trying to use the new connector "File Connector" to write on parquet file. i created simple job takes from teradata as a source and write on the parquet file as a target.
Image of the job
but facing below error :
> File_Connector_20,0: java.lang.NoClassDefFoundError: org.apache.hadoop.fs.FileSystem
at java.lang.J9VMInternals.prepareClassImpl (J9VMInternals.java)
at java.lang.J9VMInternals.prepare (J9VMInternals.java: 304)
at java.lang.Class.getConstructor (Class.java: 594)
at com.ibm.iis.jis.utilities.dochandler.impl.OutputBuilder.<init> (OutputBuilder.java: 80)
at com.ibm.iis.jis.utilities.dochandler.impl.Registrar.getBuilder (Registrar.java: 340)
at com.ibm.iis.jis.utilities.dochandler.impl.Registrar.getBuilder (Registrar.java: 302)
at com.ibm.iis.cc.filesystem.FileSystem.getBuilder (FileSystem.java: 2586)
at com.ibm.iis.cc.filesystem.FileSystem.writeFile (FileSystem.java: 1063)
at com.ibm.iis.cc.filesystem.FileSystem.process (FileSystem.java: 935)
at com.ibm.is.cc.javastage.connector.CC_JavaAdapter.run (CC_JavaAdapter.java: 444)
i followed the steps in below link :
https://www.ibm.com/support/knowledgecenter/SSZJPZ_11.7.0/com.ibm.swg.im.iis.conn.s3.usage.doc/topics/amaze_file_formats.html
1- i uploaded the jar files into "/ds9/IBM/InformationServer/Server/DSComponents/jars"
2- added them to CLASSPATH in agent.sh then restarted the datastage.
3- i have set The environment variable CC_USE_LATEST_FILECC_JARS to the value parquet-1.9.0.jar:orc-2.1.jar.
i tried also to add the CLASSPATH as an environment variable in the job but not worked.
noting that i'm using Local in File System.
so any hint is appreciated as i'm searching a lot time ago.
Thanks in advance,

Which File System mode you are using ? If you are using Native HDFS as File System mode, then you would need to configure CLASSPATH to include some third party jars.
Perhaps these links should provide you with some guidance.
https://www.ibm.com/support/pages/node/301847
https://www.ibm.com/support/pages/steps-required-configure-file-connector-use-parquet-or-orc-file-format
Note : Based on the hadoop distribution and version you are using, the version of the jars could be different.
If the above information does not help in resolving the issue, then you may have to reach out to IBM Support to get this addressed.

TO use File Connector, there is no need to add CLASSPATH in agent.sh unless you want to import HDFS files from IMAM.
If your requirement is reading Parquet files, then set
$CC_USE_LATEST_FILECC_JARS=parquet-1.9.0.jar
$FILECC_PARQUET_AVRO_COMPAT_MODE=TRUE
If you are still seeing issue, then run job with $CC_MSG_LEVEL=2 and open IBM support case along with job design, FULL job log and Version.xml file from Engine tier.

Get File Version of .exe in java on Linux

Question - Get File Version of .exe in java on Linux for some strange client.
Solution -
I used JNA library to read file version using Java. Given below code is running fine on windows platform but it is throwing below error on Linux docker image.
"Unable to load library 'version': Error loading shared library libversion.so: No such file or directory Error loading shared library libversion.so: No such file or directory Native library (linux-x86-64/libversion.so) not found in resource path..".
private String GetFileVersion(String filePath) {
File fileToCheck = new File(filePath);
short[] rtnData = new short[4];
int infoSize = Version.INSTANCE.GetFileVersionInfoSize(fileToCheck.getAbsolutePath(), null);
Pointer buffer = Kernel32.INSTANCE.LocalAlloc(WinBase.LMEM_ZEROINIT, infoSize);
try {
Version.INSTANCE.GetFileVersionInfo(fileToCheck.getAbsolutePath(), 0, infoSize, buffer);
IntByReference outputSize = new IntByReference();
PointerByReference pointer = new PointerByReference();
Version.INSTANCE.VerQueryValue(buffer, "\\", pointer, outputSize);
VerRsrc.VS_FIXEDFILEINFO fileInfoStructure = new VerRsrc.VS_FIXEDFILEINFO(pointer.getValue());
rtnData[0] = (short) (fileInfoStructure.dwFileVersionMS.longValue() >> 16);
rtnData[1] = (short) (fileInfoStructure.dwFileVersionMS.longValue() & 0xffff);
rtnData[2] = (short) (fileInfoStructure.dwFileVersionLS.longValue() >> 16);
rtnData[3] = (short) (fileInfoStructure.dwFileVersionLS.longValue() & 0xffff);
return String.format("%s.%s.%s.%s", rtnData[0], rtnData[1], rtnData[2], rtnData[3]);
} catch (Exception exception) {
return null;
} finally {
Kernel32.INSTANCE.GlobalFree(buffer);
}
}

I will start by answering the question that you asked, though I doubt it is what you actually need to know.
The types of different executable file formats are encoded in the first few bytes of the file. For example, ELF files (executables, shared libraries) are described in this Wikipedia page.
So there are a number of ways to find out what kind of executable in Java:
Write some code that reads the first few bytes and decodes the file header information, as per the format described in the Wikipedia link above.
Find an existing Java library that does this and work out how to do this. (Google for "java file magic library" and see what you can find.)
Read about the Linux file command and write some Java code to run file on each library and parse the output.
What I think you actually need to do is a bit different:
Locate the file or files in the file system that the Java is looking for: apparently libversion.so or linux-x86-64/libversion.so. (The file could well be a symlink. Follow it.)
Run file on each file to check that it is the right kind of library. They need to be 32 or 64 bit corresponding the JVM you are running, and the correct ABI and ISA for the platform.
Check that the files are where the JVM expects to find them. The JVM searches for libraries in directories listed in the "java.library.path" system property. You can (if necessary) set the path using a -Djava.library.path=... JVM option.
See "java.library.path – What is it and how to use" for more information on library loading.
(There is absolutely no need to do step 2 "from" or "in" Java.)

I think I have finally worked out what you are doing.
The Version you are using is actually coming from the package com.sun.jna.platform.win32. It is not part of the JNA library (jna.jar). I think it is actually part of jna-platform.jar. If I understand things correctly, that is the generated JNA adapter library for the Windows COM dlls.
If I have that correct, you would actually need the Windows COM native libraries compiled and built for the Linux platform to do what you are trying to do.
AFAIK, that's not possible.
So how could you make this work? Basically you need to do one of the following:
Look for an existing pure Java library for extracting the version information from a Windows ".exe" file. I don't think it is likely that you will find one.
Find the specification for the Windows ".exe" file format and write your own Java code to extract the version information. I haven't looked for the spec to see how much work it would be.
Then you rewrite the code that you added your question to use the alternative API.
The "libversion" file that I mentioned in my other answer is not relevant. It is something else. It is a red herring.

Internal path mapping in windows 7

I have a windows7 machine in which i am creating a file through java code in following path
C:\Users\123\Application Data\XYZ
but physically it is getting created here
C:\Users\123\AppData\Roaming\XYZ
please tell me why?

Because one is a legacy path, and the other is the current preferred folder path. Both lead to the same location.
See e.g. Application Compatibility: Junction Points and Backup Applications:
In Windows Vista® and Windows Server® 2008, the default location of user data has changed. An example of this change is the Documents and Settings directory, which has been moved from %systemdrive%\Documents and Settings to %systemdrive%\Users.
and:
..\Documents and Settings\\Application Data\ [is mapped to] ..\Users\\AppData\Roaming

How to configure MySQL JDBC driver mysql-connector-java-5.1.12?

I have downloaded MySQL JDBC driver from http://www.mysql.com/downloads/connector/j/. How do I configure it?

To the point, you just need to put it in the program's runtime classpath and then load it as follows:
try {
Class.forName("com.mysql.jdbc.Driver");
} catch (ClassNotFoundException e) {
throw new RuntimeException("Cannot find the driver in the classpath!", e);
}
If you're using an IDE, then you need to add the library to the project's Build Path. If you're running it using java.exe, then you need to specify the full path to the JDBC driver JAR file in the -cp argument, e.g. (Windows-targeted):
java -cp .;/path/to/mysql-connector-java-5.1.12.jar com.example.YourClass
For more information and hints check this small MySQL+JDBC kickoff tutorial.
Update: As per the comments, you're using Flex and you apparently want to interact with the DB at the server using Flex. Well, Flex runs at the client machine and Java + the DB runs at the server machine. Both are connected by network with HTTP as communication protocol. You need to write Java code on the server side accordingly (Servlet? Webservice?) which interacts with the DB based on the parameters/pathinfo given with the HTTP request and returns the desired results. Then you can just invoke HTTP requests from inside Flex and process the HTTP response.

You can follow the guidelines given at: https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-installing.html.
Also check the comments for more ideas and info. I.e: putting it in [PATH_TO_JAVA]/jre/lib/ext/ (on Win Mashine) or /Library/Java/Extensions (Mac OS X) etc.

Refer < http://www.developer.com/java/data/jdbc-and-mysql-installation-and-preparation-of-mysql.html> or
The JDBC .jar file needs to be added to the library, this can be done by adding it manually to '...jre\lib\ext' folder of your Java installation. It will be automatically included in the default library available to every project you create.

How can I write System preferences with Java? Can I invoke UAC?

How can I write system preferences with Java, using Preferences.systemRoot()?
I tried with:
Preferences preferences = Preferences.systemRoot();
preferences.put("/myapplication/databasepath", pathToDatabase);
But I got this error message:
2010-maj-29 19:02:50 java.util.prefs.WindowsPreferences openKey
VARNING: Could not open windows registry node Software\JavaSoft\Prefs at root 0x80000002. Windows RegOpenKey(...) returned error code 5.
Exception in thread "AWT-EventQueue-0" java.lang.SecurityException: Could not open windows registry node Software\JavaSoft\Prefs at root 0x80000002: Access denied
at java.util.prefs.WindowsPreferences.openKey(Unknown Source)
at java.util.prefs.WindowsPreferences.openKey(Unknown Source)
at java.util.prefs.WindowsPreferences.openKey(Unknown Source)
at java.util.prefs.WindowsPreferences.putSpi(Unknown Source)
at java.util.prefs.AbstractPreferences.put(Unknown Source)
at org.example.install.Setup$2.actionPerformed(Setup.java:43)
I would like to do this, because I want to install an embedded JavaDB database, and let multiple users on the computer to use the same database with the application.
How to solve this? Can I invoke UAC and do this as Administrator from Java? And if I log in as Administrator when writing, can I read the values with my Java application if I'm logged in as a User?

You cannot write to any arbitrary registry location from java preferences - all preferences are stored under a subkey Software\Javasoft\Prefs. With user preferences mapping to the HKEY_CURRENT_USER hive, and system mapping to the HKEY_LOCAL_MACHINE hive.
To write to the registry, you could use the windows "REG" command line tool. This page details other ways of modifying the registry. including use of .reg files.
I had the same need - to write to the registry from java - I solved it by writing a small .NET command line utility to do it.
The Sun Windows JDK does ship with generic code to write to arbitrary portions of the registry (WindowsPreferences), but it's not public. This article describes how to access this class using reflection.

You can't edit Preferences.systemRoot() if User Account Control is turned on. Seems like Microsoft went and broke it. There is a workaround here, but it's not straightforward.

So I had this same issue, so I opened an issue with Oracle:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7043176
I was able to work around it myself, by writing a custom implementation of AbstractPreferences and a corresponding PreferencesFactory. What I did was on Windows have the system preferences write to the application data directory defined in the registry by:
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders\Common AppData
I used Runtime.getRuntime().exec("reg query \""+key+ "\" /v \""+value+"\"") to get that (works even with UAC turned on).
That evaluates to "C:\ProgramData" on Windows 7 and "C:\Documents and Settings\All Users\Application Data" on XP. I added a subdirectory called "JavaPreferences" and wrote an implementation that uses a properties file as the backend.
As a side note, I had a similar issue with system preferences on Linux because the installer for the JRE was not run by root so I didn't have access to "/etc/.java". A ended up picking another custom directory and granting permissions for that.