We Keep Coding

Error while trying to write on parquet file in datastage 11.7 (File_Connector_20,0: java.lang.NoClassDefFoundError: org.apache.hadoop.fs.FileSystem)

we have recently upgraded the DataStage from 9.1 to 11.7 on Server AIX 7.1 .
and i'm trying to use the new connector "File Connector" to write on parquet file. i created simple job takes from teradata as a source and write on the parquet file as a target.
Image of the job
but facing below error :
> File_Connector_20,0: java.lang.NoClassDefFoundError: org.apache.hadoop.fs.FileSystem
at java.lang.J9VMInternals.prepareClassImpl (J9VMInternals.java)
at java.lang.J9VMInternals.prepare (J9VMInternals.java: 304)
at java.lang.Class.getConstructor (Class.java: 594)
at com.ibm.iis.jis.utilities.dochandler.impl.OutputBuilder.<init> (OutputBuilder.java: 80)
at com.ibm.iis.jis.utilities.dochandler.impl.Registrar.getBuilder (Registrar.java: 340)
at com.ibm.iis.jis.utilities.dochandler.impl.Registrar.getBuilder (Registrar.java: 302)
at com.ibm.iis.cc.filesystem.FileSystem.getBuilder (FileSystem.java: 2586)
at com.ibm.iis.cc.filesystem.FileSystem.writeFile (FileSystem.java: 1063)
at com.ibm.iis.cc.filesystem.FileSystem.process (FileSystem.java: 935)
at com.ibm.is.cc.javastage.connector.CC_JavaAdapter.run (CC_JavaAdapter.java: 444)
i followed the steps in below link :
https://www.ibm.com/support/knowledgecenter/SSZJPZ_11.7.0/com.ibm.swg.im.iis.conn.s3.usage.doc/topics/amaze_file_formats.html
1- i uploaded the jar files into "/ds9/IBM/InformationServer/Server/DSComponents/jars"
2- added them to CLASSPATH in agent.sh then restarted the datastage.
3- i have set The environment variable CC_USE_LATEST_FILECC_JARS to the value parquet-1.9.0.jar:orc-2.1.jar.
i tried also to add the CLASSPATH as an environment variable in the job but not worked.
noting that i'm using Local in File System.
so any hint is appreciated as i'm searching a lot time ago.
Thanks in advance,

Which File System mode you are using ? If you are using Native HDFS as File System mode, then you would need to configure CLASSPATH to include some third party jars.
Perhaps these links should provide you with some guidance.
https://www.ibm.com/support/pages/node/301847
https://www.ibm.com/support/pages/steps-required-configure-file-connector-use-parquet-or-orc-file-format
Note : Based on the hadoop distribution and version you are using, the version of the jars could be different.
If the above information does not help in resolving the issue, then you may have to reach out to IBM Support to get this addressed.

TO use File Connector, there is no need to add CLASSPATH in agent.sh unless you want to import HDFS files from IMAM.
If your requirement is reading Parquet files, then set
$CC_USE_LATEST_FILECC_JARS=parquet-1.9.0.jar
$FILECC_PARQUET_AVRO_COMPAT_MODE=TRUE
If you are still seeing issue, then run job with $CC_MSG_LEVEL=2 and open IBM support case along with job design, FULL job log and Version.xml file from Engine tier.

The Tomcat installation directory is not valid. It is missing expected file or folder tcruntime-ctl.sh

Even after searching too much i couldn't find any suitable answer for my problem.
I am using Spring Tool Suite and trying to create a new server but getting error as The specified server is not valid. The .tc-runtime-instance file is missing.
So, I tried updating pivotal using Windows> Preferences and tried editing pivotal with new release but got the error as attached:
I tried changing tomcat version from 9 to 7 too but still no success. Please help.

It looks like you are trying to use the server adapter for the "Pivotal tc Server" to configure your server in the IDE. This server adapter is for the Pivotal tc Server only and doesn't work with a plain Tomcat installation on your disc. For that, you should select the "Apache Tomcat" server adapter when creating a server instance in your IDE.

heroku nodejs app - events.js:167 Error Unhandled 'error' event : spawn java ENOENT

I'm having a node-js app on Heroku using the pdfMerge.js library.
following the documentation I'm using the stream event mechanism as a callback to identify the end of the process
then an exception is thrown :
events.js:167 Error: spawn java ENOENT.
I'm almost sure it's happening because I'm messing required java installation as described here:
pdfmerger combines multiple PDF-files into a single PDF-file. It is a node module that utilizes the Apache PDFBox Library, which the required functionality are distributed along with this module. The only requirement for this module to run, is having Java 6 or higher in the path.
I'm Not familiar enough with Heroku installation/configuration process in order to make it work.
thanks in advance

You can add Java to your app by adding the heroku/jvm buildpack like this:
$ heroku buildpacks:add -i 1 heroku/jvm
Then redeploy with git commit --allow-empty and git push heroku master.

Error creating OpenShift application in Eclipse

I am attempting to create/deploy an application onto the Redhat OpenShift Express platform, and am attempting to do this via eclipse using the JBoss OpenShift plugins.
I am following the instructions here: https://www.redhat.com/openshift/community/blogs/getting-started-with-the-openshift-eclipse-plug-in-for-java-applications-on-the-cloud
But when entering my user credentials I get an error preventing me getting any further:
Could not check user credentials: Could not unmarshall response "
{"data":
{"user_info":
{"env_vars":null,
"max_gears":5,
"system_ssh_keys":null,
"uuid":"5a81de3405fe4ac6b5718dbb8540736f",
"rhc_domain":"rhcloud.com",
"rhlogin":"robert.hinds#accenture.com",
"ssh_keys":
{"laptop":{"type":"ssh-rsa","key":"AAAAB3NzaC1yc2EAAAABIwAAAQEAw5GIjtI+B42SBv2kmV32OHI5tCBO7JWmlhF7w9dNy2Tp7MNtK8nE7nLUKYyXSZRzK0i4azPuMDDmmJ4jFpcyAwRdzBM43xZLo/6MlMFneT3t66ukNMNOgYptiRH7Zy8Tmjquwu2dXpui77tRQaEpq4Ms/JxHvanD+YMDNTXlrv8gIDwOjhtetuyeKbf/sr+UK1Bit8eDTgai01UmjXm+u/0Hi1tD6oG/GwNviQKb/2gmrmEs0uPhcJ9Olzrn67Gyu7c2PFOjTEvOXMQHjSUrVp905t8SzffiJ7uWzVnvufrweC5gXlR/v1tp3FrFgRcZVmZX6QlhyOiy5Tr1cLD8pw=="}},
"consumed_gears":0,
"vip":false,
"namespace":"star",
"ssh_key":null,
"ssh_type":null
},
"app_info":{}
},
"api_c":["placeholder"],
"messages":"",
"broker_c":["namespace","rhlogin","ssh","app_uuid","debug","alter","cartridge","cart_type","action","app_name","api"],
"exit_code":0,
"debug":"",
"api":"1.1.3",
"result":""
}
has anyone come across this previously? I am on Win7 Eclipse Indigo on JDK7
Update:
I get this problem using the following versions/steps
Using Eclipse: Eclipse Java EE IDE for Web Developers. Version: Indigo Service Release 2 Build id: 20120216-1857
Installed the OpenShift express tools from the JBoss software update site here: http://download.jboss.org/jbosstools/updates/development/indigo/ today (the current version that is downloaded today!)
Then simply file > new > openshift aplication, then enter my username and password (created today with OpenShift Express) and I get the error.
hope thats helpful - will wait for the update to the software.

this is a "known" issue in the sense that you're not the first that reported this:
https://issues.jboss.org/browse/JBIDE-11431
The "interesting" part is that unmarshalling the response fails because of the following 2 properties with null values:
"ssh_key":null,
"ssh_type":null
This error occurrs since OpenShift added support for several keys. We fixed the unmarhslling issue but still dont know how to reproduce it. If you can we'd be very happy to get your instructions.
The fixed version of our client plugin will make it to our nightly update site in a few days. Stay tuned!

Working with Php-Java Bridge

I am having trouble setting up the Php-Java Bridge setup properly.
I will explain what I have done.
My site is in pure php
For our payment transaction process we need to set up a php-java bridge
I followed this link to setup the bridge PHP-JAVA BRIDGE INSTALATION.
Here I learned that I need to have a private jvm to install the bridge.
So 1st i installed apache-tomcat-6.0.14 in Private JVM using my c-panel. After instalation it asked me to Map a domain to private JVM. So I mapped my domain example.com (which is the only option available) to it.
Then it asked to enable a traffic redirection from Apache web server to my Java application server (there was a check box and i clicked it)
Finally it asked me to deploy the WAR File (JavaBridge.WAR was my file) and everthing seems fine
Now when i go to http://example.com/JavaBridge/ I could see the javabridge examples and it works fine.
SO FAR SO GOOD
Now my problem starts here when I try to access a java class file from php. A sample test.php is what I create and put the following code into it.
<?php
require_once("http://example.com:portnumber/JavaBridge/java/Java.inc");
$System = java("java.lang.System");
echo $System->getProperties(); //This Part echo's correctly and shows the data so it means i can access Java.inc Correctly
$path_e24class = getcwd(). '/e24PaymentPipe.class'; //This part fails both test.php and java class file e24PaymentPipe.class are in the same directory in publich_html folder
java_require($path_e24class);
$pipe = new Java("e24PaymentPipe");
$pipe->setAction("1");
?>
My site contents reside in the public_html folder and the WAR file are deployed in private jvm.
These are the error message am getting.
1) Warning: java_require() not supported anymore. Please use tomcat or jee hot deployment instead
Fatal error: Uncaught [[o:Exception]:"java.lang.Exception: CreateInstance failed: new e24PaymentPipe. Cause: java.lang.ClassNotFoundException: e24PaymentPipe VM: 1.6.0_22#http://java.sun.com/" at: #-10
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1358) #-9
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204) #-8
java.lang.Class.forName0(Native Method) #-7
java.lang.Class.forName(Class.java:247) #-6
php.java.bridge.Util.classForName(Util.java:1518) #-5
php.java.bridge.JavaBridge.CreateObject(JavaBridge.java:445) #-4
php.java.bridge.Request.handleRequest(Request.java:458) #-3
php.java.bridge.Request.handleRequests(Request.java:500) #-2
php.java.bridge.http.ContextRunner.run(ContextRunner.java:145) #-1
php.java.bridge.ThreadPool$Delegate.run(ThreadPool.java:60) #0
http://example.com:portnumber/JavaBridge/java/Java.inc(232): java_ThrowExceptionProxyFactory->getProxy(3, 'java.util.Prope...', 'T', false) #1
Finally I don't know much about the java. So am stuck here not knowing what to do.

Here is a great step by step tutorial you can follow, which shows everything required! It is a little old (2007) but helped me a while ago.
There is also another option. You can install Apache Tomcat and deploy your war there. You can have even multiple tomcat instances simultaneously with your httpd running at the same time on the same machine, as long as you respect the port settings. You can even front them with Apache httpd.

you can try this:
package your code to jar, and copy it to java.ext.dirs which you can found in JavaBridge.log
copy the related class libraries to java.ext.dirs
restart the service of JavaBridge
good luck!
<?php require_once("JavaBridge/java/Java.inc");
try {
$hd = new java("hdfs.HDFS");
$hd->get("hdfs://master:9000/user/hadoop/test-in/logo_cn.png", "/home/hadoop/1.png");
} catch (JavaException $ex) { echo "An exception occured: "; echo $ex; echo "<br>\n";}
?>

You can use this php implementation on github that works with php 5.3.
See credits on the git readme for more information.

You can try this; put the JavaBridge.jar in tomcat's lib folder e.g. apache-tomcat-7.0.12/lib.
Restart tomcat server and then,
$pipe = new java("com.aciworldwide.commerce.gateway.plugins.e24PaymentPipe");
$pipe->setAction("1");
This way I created the php version of the object.

Why don't you put the e24PaymentPipe class in your Java application's classpath and skip the two lines below:
// $path_e24class = getcwd(). '/e24PaymentPipe.class';
// java_require($path_e24class);
$pipe = new java("fully.qualified.classpath.e24PaymentPipe");
You are mixing PHP side and Java side operations. in theory the java_require (which is deprecated) was designed to work on the Java side. You are specifying a PHP side path.

You can save yourself a lot of grief by using a pure PHP implementation of the e24PaymentPipe library.
Disclaimer
The link is to my github repo of the library, but I did not write it. See the readme in for original credits.