We Keep Coding

Error while trying to write on parquet file in datastage 11.7 (File_Connector_20,0: java.lang.NoClassDefFoundError: org.apache.hadoop.fs.FileSystem)

we have recently upgraded the DataStage from 9.1 to 11.7 on Server AIX 7.1 .
and i'm trying to use the new connector "File Connector" to write on parquet file. i created simple job takes from teradata as a source and write on the parquet file as a target.
Image of the job
but facing below error :
> File_Connector_20,0: java.lang.NoClassDefFoundError: org.apache.hadoop.fs.FileSystem
at java.lang.J9VMInternals.prepareClassImpl (J9VMInternals.java)
at java.lang.J9VMInternals.prepare (J9VMInternals.java: 304)
at java.lang.Class.getConstructor (Class.java: 594)
at com.ibm.iis.jis.utilities.dochandler.impl.OutputBuilder.<init> (OutputBuilder.java: 80)
at com.ibm.iis.jis.utilities.dochandler.impl.Registrar.getBuilder (Registrar.java: 340)
at com.ibm.iis.jis.utilities.dochandler.impl.Registrar.getBuilder (Registrar.java: 302)
at com.ibm.iis.cc.filesystem.FileSystem.getBuilder (FileSystem.java: 2586)
at com.ibm.iis.cc.filesystem.FileSystem.writeFile (FileSystem.java: 1063)
at com.ibm.iis.cc.filesystem.FileSystem.process (FileSystem.java: 935)
at com.ibm.is.cc.javastage.connector.CC_JavaAdapter.run (CC_JavaAdapter.java: 444)
i followed the steps in below link :
https://www.ibm.com/support/knowledgecenter/SSZJPZ_11.7.0/com.ibm.swg.im.iis.conn.s3.usage.doc/topics/amaze_file_formats.html
1- i uploaded the jar files into "/ds9/IBM/InformationServer/Server/DSComponents/jars"
2- added them to CLASSPATH in agent.sh then restarted the datastage.
3- i have set The environment variable CC_USE_LATEST_FILECC_JARS to the value parquet-1.9.0.jar:orc-2.1.jar.
i tried also to add the CLASSPATH as an environment variable in the job but not worked.
noting that i'm using Local in File System.
so any hint is appreciated as i'm searching a lot time ago.
Thanks in advance,

Which File System mode you are using ? If you are using Native HDFS as File System mode, then you would need to configure CLASSPATH to include some third party jars.
Perhaps these links should provide you with some guidance.
https://www.ibm.com/support/pages/node/301847
https://www.ibm.com/support/pages/steps-required-configure-file-connector-use-parquet-or-orc-file-format
Note : Based on the hadoop distribution and version you are using, the version of the jars could be different.
If the above information does not help in resolving the issue, then you may have to reach out to IBM Support to get this addressed.

TO use File Connector, there is no need to add CLASSPATH in agent.sh unless you want to import HDFS files from IMAM.
If your requirement is reading Parquet files, then set
$CC_USE_LATEST_FILECC_JARS=parquet-1.9.0.jar
$FILECC_PARQUET_AVRO_COMPAT_MODE=TRUE
If you are still seeing issue, then run job with $CC_MSG_LEVEL=2 and open IBM support case along with job design, FULL job log and Version.xml file from Engine tier.

Issue with the name of cache directory in Android

In my app I am saving temporary files to my app cache folder. I am getting the name of the cache directory via this method:
context.getCacheDir();
It returns me such path: /data/user/0/my.app.packagename
But later when I am trying to get name of parent directory of my cache file via this method - file.getParent(); I am getting absolutely different path to cache directory, in my case this: /data/data/my.app-packagename
So I am just wondering why this is happening, why getParent() doesn't return the same path as context.getCacheDir()?

The behaviour is correct, technically its the same.
If you connect to your android device via adb shell and go into /data/user/ and run the 'll' command you will see that the folder 0 is just a symbolic link to /data/data/.
root#android:/data/user # ll
lrwxrwxrwx root root 1970-01-01 01:00 0 -> /data/data/
If you dont know what that means, read up on symbolic links.
So there is no issue and you can trust android that they are the same.

This behavior confused me a lot today.
Here are my 50 Cent to bring light into this:
I've started an Emulator with Android 8.1
I've connected my Android 9 Device via USB
After that I've started Android-Studio and opend the Device File Explorer.
Now strange things happend:
On Android 8.1 you see the structure /data/user/0/com.myApp/
on Android 9 there is no directory /data/user (and no symbolic link visible via android-studio)
BUT, internally the path seems to be the same (even if there is no symbolic Link visible in android studio).
I use RNFetchBlob (React native package to read files), and if I output the Result from RNFetchBlob.fs.dirs.CacheDir the result was /data/user/0/.... even if on Android 9 this Directory not exist.
But I'd gave it a try and output all files that will be found at this Directory with the following Command :
RNFetchBlob.fs.ls(RNFetchBlob.fs.dirs.CacheDir)
.then((files) => {
console.log("output-files",files);
})
...and found that this output all the files I've cached before.
If anybody knows some documentation why this behavior exist up from Android 9, please comment and point us to this.

Matlab installation (LD_LIBRARY_PATH) messes up other library files

I am trying to install Matlab on a Linux machine, but setting LD_LIBRARY_PATH (as the installation requires) breaks other library files. I am not an Linux expert, but I have tried several things and cannot get it working correctly. I have even contacted Matlab support, got the issue elevated to the dev team, and was basically told "haha sucks to suck". I have seen a few other people online have had the same issue, but either their questions were never answered or they had a slightly different problem and their solution didn't apply to me.
Installing on a VM running Ubuntu:
I set LD_LIBRARY_PATH as the instructions say, then it breaks network files. I can ping google.com, but I cannot nslookup google.com or visit it in a browser. Nslookup provides this error:
nslookup: /usr/local/MATLAB/MATLAB_Runtime/v90/bin/glnxa64/libcrypto.so.1.0.0: no version information available (required by /usr/lib/libdns.so.100)
03-Feb-2016 11:32:22.361 ENGINE_by_id failed (crypto failure)
03-Feb-2016 11:32:22.362 error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:244:
03-Feb-2016 11:32:22.363 error:260B6084:engine routines:DYNAMIC_LOAD:dso not found:eng_dyn.c:447:
03-Feb-2016 11:32:22.363 error:2606A074:engine routines:ENGINE_by_id:no such engine:eng_list.c:418:id=gost
(null): dst_lib_init: crypto failure
The installation worked though (I can run my Java programs that reference compiled Matlab functions). Unsetting LD_LIBRARY_PATH fixes the network files but then I can't run programs anymore.
Installing on EC2 instance:
On an EC2 instance it does not break the network files (nslookup is fine). Instead it messes up Python library files. Trying to use any aws cli command, I get the error:
File "/usr/bin/aws", line 19, in <module>
import awscli.clidriver
File "/usr/lib/python2.7/dist-packages/awscli/clidriver.py", line 16, in <module>
import botocore.session
File "/usr/lib/python2.7/dist-packages/botocore/session.py", line 25, in <module>
import botocore.config
File "/usr/lib/python2.7/dist-packages/botocore/config.py", line 18, in <module>
from botocore.compat import six
File "/usr/lib/python2.7/dist-packages/botocore/compat.py", line 139, in <module>
import xml.etree.cElementTree
File "/usr/lib64/python2.7/xml/etree/cElementTree.py", line 3, in <module>
from _elementtree import *
ImportError: PyCapsule_Import could not import module "pyexpat"
Printing sys.path in Python shows lib-dynload is already there though, so it doesn't seem to the problem.
And when trying to run the program, I get:
Exception in thread "main" java.lang.LinkageError: libXt.so.6: cannot open shared object file: No such file or directory
at com.mathworks.toolbox.javabuilder.internal.DynamicLibraryUtils.dlopen(Native Method)
at com.mathworks.toolbox.javabuilder.internal.DynamicLibraryUtils.loadLibraryAndBindNativeMethods(DynamicLibraryUtils.java:134)
at com.mathworks.toolbox.javabuilder.internal.MWMCR.<clinit>(MWMCR.java:1529)
at VectorAddExample.VectorAddExampleMCRFactory.newInstance(VectorAddExampleMCRFactory.java:48)
at VectorAddExample.VectorAddExampleMCRFactory.newInstance(VectorAddExampleMCRFactory.java:59)
at VectorAddExample.VectorAddClass.<init>(VectorAddClass.java:62)
at com.mypackage.Example.main(Example.java:13)
I'm at a brick wall and really have no clue how to proceed.

Maybe something else already needs LD_LIBRARY_PATH set to work. Make sure you prepend not overwrite:
export LD_LIBRARY_PATH=new/path:$LD_LIBRARY_PATH
Edit:
OK, if LD_LIBRARY_PATH was initially empty, this suggests that Matlab comes with shared libraries that are incompatible with your system ones:
nslookup: /usr/local/MATLAB/MATLAB_Runtime/v90/bin/glnxa64/libcrypto.so.1.0.0: no version information available (required by /usr/lib/libdns.so.100)
suggests that /usr/lib/libdns.so.100 needs libcrypto.so.1.0.0, which is now being resolved to the one that comes with MATLAB, which is incompatible.
You can check the dependencies of a dll by
ldd /usr/lib/libcrypto.so.1.0.0
and hopefully you can find a configuration that keeps both MATLAB and your system happy. Unfortunately, this may involve a lot of trial and error.
If there is no such configuration, you can try setting LD_LIBRARY_PATH only when you run MATLAB:
LD_LIBRARY_PATH=$MATLAB_LD_LIBRARY_PATH matlab
Edit 2:
Well, for the Python issue, it seems to boil down to pyexpat, which is a wrapper around the standard expat XML parser. Try doing (name guessed since I don't have a Linux right now):
ldd /usr/local/lib/python2.7/site-packages/libpyexpat.so
and see what that depends on. Probably, it will be libexpat.so, which is now being resolved to MATLAB's version.

try the following command:
export LD_LIBRARY_PATH=/usr/local/MATLAB/MATLAB_Runtime/v90/runtime/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v90/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v90/sys/os‌​/glnxa64:$LD_LIBRARY_PATH

Perhaps not helpful for OP but if you are generating a python package with MATLAB, you could modify the generated __init__.py file MATLAB creates for your package.
Specifically, the generated __init__.py file contains the following line (as of MATLAB 2017a):
PLATFORM_DICT = {'Windows': ['PATH','dll',''], 'Linux': ['LD_LIBRARY_PATH','so','libmw'], 'Darwin': ['DYMCR_LIBRARY_PATH','dylib','libmw']}
For Linux platform, you could simply replace LD_LIBRARY_PATH with something else such as MCR_LIBRARY_PATH to prevent mucking with your shared libs.
sed -i -e 's/LD_LIBRARY_PATH/MCR_LIBRARY_PATH/g' /MY/PACKAGE/BUILD/PATH/__init__.py
Then obviously export MCR_LIBRARY_PATH before using python.

JAVA EE 7 First cup tutorial, can't install Glassfish 4 update center

Well I was trying to follow the first cup tutorial, but I'm stuck with a Glassfish error.
I'm supposed to this:
1.2.4 Getting the Latest Updates to the Tutorial
Check for any updates to this tutorial by using the Update Center
included with the Java EE 7 SDK.
1.2.4.1 Update the Tutorial Through the Update Center Open the Update Center and check for any updates to the tutorial.
In NetBeans IDE, select the Services tab and expand the Servers node.
Right-click the GlassFish Server instance and select View Update
Center to display the Update Tool. In the tree, select Available
Updates to display a list of updated packages. Look for updates to the
First Cup for Java EE 7 (javaee-firstcup-tutorial) package. If there
is an updated version of First Cup, select First Cup 7.0 for Java EE 7
(javaee-firstcup-tutorial) and click Install.
The thing is, when I try to open the Update Center, this is what I get:
The software needed for this command (updatetool) is not installed.
If you choose to install Update Tool, your system will be automatically
configured to periodically check for software updates. If you would like
to configure the tool to not check for updates, you can override the
default behavior via the tool's Preferences facility.
Exception in thread "main" java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:658)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at com.sun.pkg.client.Image.checkRepositoryConnection(Image.java:1225)
at com.sun.pkg.client.Catalog.refresh(Catalog.java:132)
at com.sun.pkg.client.Image.refreshCatalogs(Image.java:1627)
at com.sun.pkg.client.Client.main(Client.java:109)
When this tool interacts with package repositories, some system information
such as your system's IP address and operating system type and version
is sent to the repository server. For more information please see:
http://wikis.oracle.com/display/updatecenter/UsageMetricsUC2
Once installation is complete you may re-run this command.
Would you like to install Update Tool now (y/n):
C:\Program Files\glassfish-4.0>"C:\Program Files\Java\jdk1.7.0_40\bin\java" -Dimage.path="C:\Program Files\glassfish-4.0\bin\\.." -jar "C:\Program Files\glassfish-4.0\bin\\..\pkg/lib/pkg-client.jar" refresh
C:\Program Files\glassfish-4.0>"C:\Program Files\Java\jdk1.7.0_40\bin\java" -Dimage.path="C:\Program Files\glassfish-4.0\bin\\.." -jar "C:\Program Files\glassfish-4.0\bin\\..\pkg/lib/pkg-bootstrap.jar" "C:\Users\Fabio\AppData\Local\Temp\pkg-bootstrap11073.props"
Proxy: Using system proxy settings.
Input/output error: Connection reset
Could not download application packages. This could be because:
- a proxy server is needed to access the internet. Please ensure that
the system proxy server settings in your Internet Options control panel
(under Connections:LAN Settings) are correct, or set the HTTP_PROXY
environment variable to the full URL of the proxy server.
- the package server or network connection is slow.
If you are getting time out errors you can try setting the
PKG_CLIENT_CONNECT_TIMEOUT and PKG_CLIENT_READ_TIMEOUT
environment variables and try again. For example to increase
the timeouts to 300 seconds set them to 300
- the package server is down or otherwise inaccessible or it is
generating invalid data. Please contact the provider of the package
server.

This may be a problem with the slow GlassFish update server. Here are some steps you can try:
You should retry first to see if the error occurs again. You can start a cmd, navigate to the \glassfish\bin directory and run updatetool.bat so you can see the output which is printed after the one you posted.
If you are behind a proxy you should make sure it is setup correctly in the system settings.
You can also try to install the updates through the GlassFish admin console. Start your GlassFish instance and navigate to http:\\localhost:4848. On the bottom of the left menu click on Updatetool. There you can choose and install the desired updates.
Another option is to change the timeouts of the pkg tool. To set the timeout to 300 seconds do the following in a cmd (on Windows):
set PKG_CLIENT_CONNECT_TIMEOUT=300
set PKG_CLIENT_READ_TIMEOUT=300
Your last (ok, there may be other ones) option is to just skip the update process and do the tutorial with the current version. It should work anyway and I can see in my GlassFish update center that there is only an update for the tutorial documentation at the moment.

Just set this line into cmd (like administrator):
set PKG_CLIENT_CONNECT_TIMEOUT=300
set PKG_CLIENT_READ_TIMEOUT=300

Maybe the firstcup example is already existed in the install directory. Try to find it in the directory_install_glassfish/glassfish4/docs. The mine contain even javaee-tutorial and firstcup. For the next part, you maybe continue after guide in firstcup.pdf with the part of : Creating Your First Java by create archetype of java project. Good luck !

For the lazy of you the pkg Unix command is:
export PKG_CLIENT_CONNECT_TIMEOUT=300
export PKG_CLIENT_READ_TIMEOUT=300

After being stuck for a while on the installation of GlassFish update server from Netbeans (in my company network, proxy issues), I got it this way:
- Advanced System Settings > System Properties window > Advanced tab, New System Variable: http_proxy, http://username:password#your_proxy:your_port
- Netbeans (running as admin just in case), Services > Server > Glassfish Server > Right button View Domain Update Center, then install
It worked for me, Hope it helps!

Working with Php-Java Bridge

I am having trouble setting up the Php-Java Bridge setup properly.
I will explain what I have done.
My site is in pure php
For our payment transaction process we need to set up a php-java bridge
I followed this link to setup the bridge PHP-JAVA BRIDGE INSTALATION.
Here I learned that I need to have a private jvm to install the bridge.
So 1st i installed apache-tomcat-6.0.14 in Private JVM using my c-panel. After instalation it asked me to Map a domain to private JVM. So I mapped my domain example.com (which is the only option available) to it.
Then it asked to enable a traffic redirection from Apache web server to my Java application server (there was a check box and i clicked it)
Finally it asked me to deploy the WAR File (JavaBridge.WAR was my file) and everthing seems fine
Now when i go to http://example.com/JavaBridge/ I could see the javabridge examples and it works fine.
SO FAR SO GOOD
Now my problem starts here when I try to access a java class file from php. A sample test.php is what I create and put the following code into it.
<?php
require_once("http://example.com:portnumber/JavaBridge/java/Java.inc");
$System = java("java.lang.System");
echo $System->getProperties(); //This Part echo's correctly and shows the data so it means i can access Java.inc Correctly
$path_e24class = getcwd(). '/e24PaymentPipe.class'; //This part fails both test.php and java class file e24PaymentPipe.class are in the same directory in publich_html folder
java_require($path_e24class);
$pipe = new Java("e24PaymentPipe");
$pipe->setAction("1");
?>
My site contents reside in the public_html folder and the WAR file are deployed in private jvm.
These are the error message am getting.
1) Warning: java_require() not supported anymore. Please use tomcat or jee hot deployment instead
Fatal error: Uncaught [[o:Exception]:"java.lang.Exception: CreateInstance failed: new e24PaymentPipe. Cause: java.lang.ClassNotFoundException: e24PaymentPipe VM: 1.6.0_22#http://java.sun.com/" at: #-10
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1358) #-9
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204) #-8
java.lang.Class.forName0(Native Method) #-7
java.lang.Class.forName(Class.java:247) #-6
php.java.bridge.Util.classForName(Util.java:1518) #-5
php.java.bridge.JavaBridge.CreateObject(JavaBridge.java:445) #-4
php.java.bridge.Request.handleRequest(Request.java:458) #-3
php.java.bridge.Request.handleRequests(Request.java:500) #-2
php.java.bridge.http.ContextRunner.run(ContextRunner.java:145) #-1
php.java.bridge.ThreadPool$Delegate.run(ThreadPool.java:60) #0
http://example.com:portnumber/JavaBridge/java/Java.inc(232): java_ThrowExceptionProxyFactory->getProxy(3, 'java.util.Prope...', 'T', false) #1
Finally I don't know much about the java. So am stuck here not knowing what to do.

Here is a great step by step tutorial you can follow, which shows everything required! It is a little old (2007) but helped me a while ago.
There is also another option. You can install Apache Tomcat and deploy your war there. You can have even multiple tomcat instances simultaneously with your httpd running at the same time on the same machine, as long as you respect the port settings. You can even front them with Apache httpd.

you can try this:
package your code to jar, and copy it to java.ext.dirs which you can found in JavaBridge.log
copy the related class libraries to java.ext.dirs
restart the service of JavaBridge
good luck!
<?php require_once("JavaBridge/java/Java.inc");
try {
$hd = new java("hdfs.HDFS");
$hd->get("hdfs://master:9000/user/hadoop/test-in/logo_cn.png", "/home/hadoop/1.png");
} catch (JavaException $ex) { echo "An exception occured: "; echo $ex; echo "<br>\n";}
?>

You can use this php implementation on github that works with php 5.3.
See credits on the git readme for more information.

You can try this; put the JavaBridge.jar in tomcat's lib folder e.g. apache-tomcat-7.0.12/lib.
Restart tomcat server and then,
$pipe = new java("com.aciworldwide.commerce.gateway.plugins.e24PaymentPipe");
$pipe->setAction("1");
This way I created the php version of the object.

Why don't you put the e24PaymentPipe class in your Java application's classpath and skip the two lines below:
// $path_e24class = getcwd(). '/e24PaymentPipe.class';
// java_require($path_e24class);
$pipe = new java("fully.qualified.classpath.e24PaymentPipe");
You are mixing PHP side and Java side operations. in theory the java_require (which is deprecated) was designed to work on the Java side. You are specifying a PHP side path.

You can save yourself a lot of grief by using a pure PHP implementation of the e24PaymentPipe library.
Disclaimer
The link is to my github repo of the library, but I did not write it. See the readme in for original credits.