We Keep Coding

Cassandra cluster giving connection refused error when writing from hadoop using CQLOutputformat

I have a single node cassandra cluster which is NOT colocated with my hadoop cluster. I'm using this hadoop cluster to bulk load data into a Cassandra cluster using CqlOutputFormat with following job configuration.
Job job = new Job(conf);
Configuration jobConf = job.getConfiguration();
ConfigHelper.setOutputInitialAddress(jobConf, "10.27.124.73");
ConfigHelper.setInputInitialAddress(jobConf, "10.27.124.73");
//ConfigHelper.setOutputRpcPort(jobConf, "9160"); I tried setting the ports. Doesn't help.
//ConfigHelper.setInputRpcPort(jobConf, "9160");
ConfigHelper.setOutputPartitioner(jobConf,"Murmur3Partitioner");
ConfigHelper.setInputPartitioner(jobConf, "Murmur3Partitioner");
ConfigHelper.setInputColumnFamily(jobConf, "pinspace", "pinseries");
ConfigHelper.setOutputColumnFamily(jobConf, "pinspace", "pinseries");
System.out.println(ConfigHelper.getOutputColumnFamily(jobConf));
String query = "update pinspace.pinseries set timeseries = ?";
CqlConfigHelper.setOutputCql(jobConf, query);
CqlConfigHelper.setInputCQLPageRowSize(jobConf, "3");
job.setInputFormatClass(BoomInputFormat.class);
job.setMapperClass(GrepMapper.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setReducerClass(GrepReducer.class);
job.setNumReduceTasks(1024);
job.setOutputFormatClass(CqlOutputFormat.class);
I get Connection refused error with the following stacktrace:
Error: java.io.IOException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:280)
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at org.apache.cassandra.thrift.TFramedTransportFactory.openTransport(TFramedTransportFactory.java:41)
at org.apache.cassandra.hadoop.AbstractColumnFamilyOutputFormat.createAuthenticatedClient(AbstractColumnFamilyOutputFormat.java:123)
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:271)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 4 more
I'm using cassandra-driver-core 2.1.5 and cassandra-all 2.1.3.
The server is pingable using the given IP from the datanodes and telnet 10.27.124.73 9160 and telnet 10.27.124.73 9042 works.
EDIT:
I tried setting 9042 port in the code using,
ConfigHelper.setOutputRpcPort(jobConf, "9042");
Then the error changed to:
Error: java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Read a negative frame size (-2097152000)!
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter.<init>(CqlRecordWriter.java:130)
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter.<init>(CqlRecordWriter.java:88)
at org.apache.cassandra.hadoop.cql3.CqlOutputFormat.getRecordWriter(CqlOutputFormat.java:74)
at org.apache.cassandra.hadoop.cql3.CqlOutputFormat.getRecordWriter(CqlOutputFormat.java:55)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.thrift.transport.TTransportException: Read a negative frame size (-2097152000)!
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:133)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.cassandra.thrift.Cassandra$Client.recv_set_keyspace(Cassandra.java:608)
at org.apache.cassandra.thrift.Cassandra$Client.set_keyspace(Cassandra.java:595)
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter.<init>(CqlRecordWriter.java:108)
... 11 more

Getting connection time out while running Oozie workflow through Java

I am getting following error when i try to run Oozie workflow using java
IO_ERROR : java.net.ConnectException: Connection timed out: connect
at org.apache.oozie.client.OozieClient.validateWSVersion(OozieClient.java:234)
at org.apache.oozie.client.OozieClient.createURL(OozieClient.java:300)
at org.apache.oozie.client.OozieClient.access$000(OozieClient.java:71)
at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:366)
at org.apache.oozie.client.OozieClient.run(OozieClient.java:547)
at oozieDemo.main(oozieDemo.java:27)
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
Here is my code:
OozieClient wc = new OozieClient("http:xxxxxxx/oozie");
System.out.println(" connection established....." + wc);
Properties conf = wc.createConfiguration();
conf.setProperty(OozieClient.APP_PATH,"hdfs:foo/xxx/workflow.xml");
conf.setProperty("jobTracker", "foo:8021");
conf.setProperty("nameNode","hdfs:xxxx");
conf.setProperty("queueName", "default");
conf.setProperty("appLibLoc","hdfs:/foo/xxx/lib");
String jobId = wc.run(conf);
System.out.println("Workflow job submitted");
So here I can see connection is getting established but unable to run the workflow.
I am new to this. So can't figure it out where exactly it is failing.

Connection timeout means either:
URL is incorrect or down, try pinging it.
Firewall is blocking it.
Default timeout expired.
Your internet access is down, which I'm going to assume isn't the case.

How to set a custom port for RMI regisrty?

In my RMI program I want to set up my RMI registry on the port 8080, but when I do it, I get an exception.
Here is my server code
public class Server {
public static void main(String[] args) {
try {
LocateRegistry.createRegistry(8080);
MathServerImpl mathServer = new MathServerImpl();
Naming.rebind("MathServer", mathServer);
System.out.println("Math Server has started and is running");
} catch (RemoteException | MalformedURLException e) {
System.out.println(e.getMessage());
e.printStackTrace();
}
}
}
The code works properly for the port 1099, which is it's default port as far as I know, but for this case it gives me a java.net.ConnectException, and here is the log.
java.rmi.ConnectException: Connection refused to host: 10.100.25.173; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:341)
at sun.rmi.registry.RegistryImpl_Stub.rebind(Unknown Source)
at java.rmi.Naming.rebind(Naming.java:177)
at server.Server.main(Server.java:16)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:147)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 11 more

You are creating a registry on port 8080 but then attempting to rebind your object in the default registry on port 1099. You need to either:
pass a full URL to Naming.rebind including the port number, i.e. //localhost:8080/MathServer or
save a reference to the Registry object returned by LocateRegistry.createRegistry and bind the object using that registry's instance methods rather than the static methods of Naming
Similarly, you will need to use the full //localhost:8080/MathServer URL form in your clients when they lookup the object, to make sure they're talking to the right registry.

Naming.rebind("MathServer", mathServer);
Change that to
Naming.rebind("//localhost:8080/MathServer", mathServer);

If you are linux Ubuntu user then while starting RMI registry use command
rmiregistry 8080 &
For Windows
start rmiregistry 8080

Creating an RMI Connector Client

I am trying to connect to MBean server. I need to write JMX Client application. This is the code used for client application. But I have got an exception related to this
Failed to retrieve RMIServer stub:
javax.naming.ServiceUnavailableException [Root exception is
java.rmi.ConnectException: Connection refused to host: localhost;
nested exception is:
Can somebody help me to fix this.
import java.io.IOException;
import javax.management.MBeanServerConnection;
import javax.management.MBeanServerInvocationHandler;
import javax.management.MalformedObjectNameException;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
public class SystemConfigClient {
public static final String HOST = "localhost";
public static final String PORT = "1099";
public static void main(String[] args) throws IOException, MalformedObjectNameException {
JMXServiceURL url =new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + HOST + ":" +PORT+ "/jmxrmi");
JMXConnector jmxConnector = JMXConnectorFactory.connect(url);
MBeanServerConnection mbeanServerConnection = jmxConnector.getMBeanServerConnection();
//ObjectName should be same as your MBean name
ObjectName mbeanName = new ObjectName("ifs.demo1.jmx:type=SystemConfig");
//Get MBean proxy instance that will be used to make calls to registered MBean
SystemConfigMBean mbeanProxy =
(SystemConfigMBean) MBeanServerInvocationHandler.newProxyInstance(
mbeanServerConnection, mbeanName, SystemConfigMBean.class, true);
//let's make some calls to mbean through proxy and see the results.
System.out.println("Current SystemConfig::" + mbeanProxy.doConfig());
mbeanProxy.setSchemaName("NewSchema");
mbeanProxy.setThreadCount(5);
System.out.println("New SystemConfig::" + mbeanProxy.doConfig());
//let's terminate the mbean by making thread count as 0
mbeanProxy.setThreadCount(0);
//close the connection
jmxConnector.close();
}
}
I have run this code with following arguments.
Dcom.sun.management.jmxremote Dcom.sun.management.jmxremote.port=1099
Dcom.sun.management.jmxremote.authenticate=false
Dcom.sun.management.jmxremote.ssl=false
but I got the exception of
Exception in thread "main" java.io.IOException: Failed to retrieve
RMIServer stub: javax.naming.ServiceUnavailableException [Root
exception is java.rmi.ConnectException: Connection refused to host:
localhost; nested exception is: java.net.ConnectException:
Connection refused: connect] at
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
at
javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
at
javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:207)
at com.demo1.jmx.SystemConfigClient.main(SystemConfigClient.java:29)
Caused by: javax.naming.ServiceUnavailableException [Root exception is
java.rmi.ConnectException: Connection refused to host: localhost;
nested exception is: java.net.ConnectException: Connection refused:
connect] at
com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
at
com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
at javax.naming.InitialContext.lookup(InitialContext.java:392) at
javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
at
javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
at
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
... 3 more Caused by: java.rmi.ConnectException: Connection refused
to host: localhost; nested exception is: java.net.ConnectException:
Connection refused: connect at
sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601) at
sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
at
sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322) at
sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) at
com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
... 8 more Caused by: java.net.ConnectException: Connection refused:
connect at java.net.PlainSocketImpl.socketConnect(Native Method) at
java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) at
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at
java.net.Socket.connect(Socket.java:529) at
java.net.Socket.connect(Socket.java:478) at
java.net.Socket.(Socket.java:375) at
java.net.Socket.(Socket.java:189) at
sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at
sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 13 more Java Result: 1

Have you noticed that you are using the VM arguments with "D" and not "-D"? Or is it just a typo?
The correct would be:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=1099
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

DeploymentManagerCreationException

When I run the following code in command window
java weblogic.Deployer -adminurl localhost:7001 -user weblogic -password weblogic -start -name myTestDeployment"
I am getting the following error:
javax.enterprise.deploy.spi.exceptions.DeploymentManagerCreationException at
weblogic.deploy.api.spi.deploy.WebLogicDeploymentManagerImpl.<init>(W
ebLogicDeploymentManagerImpl.java:121) at
weblogic.deploy.api.spi.factories.internal.DeploymentFactoryImpl.getD
eploymentManager(DeploymentFactoryImpl.java:84) at
weblogic.deploy.api.tools.SessionHelper.getDeploymentManager(SessionH elper.java:432) at
weblogic.deploy.api.tools.deployer.Jsr88Operation.connect(Jsr88Operat ion.java:304) at
weblogic.deploy.api.tools.deployer.Deployer.perform(Deployer.java:137 ) at
weblogic.deploy.api.tools.deployer.Deployer.runBody(Deployer.java:88) at
weblogic.utils.compiler.Tool.run(Tool.java:158) at
weblogic.utils.compiler.Tool.run(Tool.java:115) at
weblogic.Deployer.run(Deployer.java:70) at weblogic.Deployer.main(Deployer.java:54)
Caused by: weblogic.deploy.api.spi.exceptions.ServerConnectionException at
weblogic.deploy.api.spi.deploy.internal.ServerConnectionImpl.init(Ser
verConnectionImpl.java:143) at
weblogic.deploy.api.spi.deploy.WebLogicDeploymentManagerImpl.getNewCo
nnection(WebLogicDeploymentManagerImpl.java:148) at
weblogic.deploy.api.spi.deploy.WebLogicDeploymentManagerImpl.<init>(W
ebLogicDeploymentManagerImpl.java:118) ... 9 more Caused by:
javax.naming.CommunicationException [Root exception is java.net.Conne ctException:
localhost:7001: Destination unreachable; nested exception is : java.net.ConnectException:
Connection refused: connect; No available rou ter to destination] at
weblogic.jndi.internal.ExceptionTranslator.toNamingException(Exceptio nTranslator.java:40)
at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLIni
tialContextFactoryDelegate.java:773) at
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLIni
tialContextFactoryDelegate.java:363) at
weblogic.jndi.Environment.getContext(Environment.java:307) at
weblogic.jndi.Environment.getContext(Environment.java:277) at
weblogic.jndi.Environment.createInitialContext(Environment.java:200) at
weblogic.jndi.Environment.getInitialContext(Environment.java:184) at
weblogic.jndi.Environment.getInitialContext(Environment.java:162) at
weblogic.deploy.api.spi.deploy.internal.ServerConnectionImpl.getConte
xt(ServerConnectionImpl.java:330) at
weblogic.deploy.api.spi.deploy.internal.ServerConnectionImpl.getEnvir
onment(ServerConnectionImpl.java:302) at
weblogic.deploy.api.spi.deploy.internal.ServerConnectionImpl.init(Ser
verConnectionImpl.java:141) ... 11 more Caused by: java.net.ConnectException:
localhost:7001: Destination unreach able; nested exception is: java.net.ConnectException:
Connection refused: connect; No available rou ter to destination at
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:204) at
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154) at
weblogic.jndi.WLInitialContextFactoryDelegate$1.run(WLInitialContextF
actoryDelegate.java:342) at
weblogic.security.acl.internal.AuthenticatedSubject.doAs(Authenticate dSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(Unknown Source) at
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLIni
tialContextFactoryDelegate.java:337) ... 19 more Caused by: java.rmi.ConnectException:
Destination unreachable; nested exception is: java.net.ConnectException: Connection
refused: connect; No available rou ter to destination at
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:472) at
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:323) at
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java :263) at
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:206) at
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226 ) at
weblogic.rjvm.RJVMFinder.findOrCreateRemoteCluster(RJVMFinder.java:30 8) at
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:194) ... 24 more Unable to connect
to 'localhost:7001': Destination unreachable; nested ex ception is:
java.net.ConnectException: Connection refused: connect; No available rou ter to
destination. Ensure the url represents a running admin server and that th e credentials
are correct. If using http protocol, tunneling must be enabled on the admin server.
Please help me to solve this problem.

Check if the admin server is running on this server port 7001:
javax.naming.CommunicationException [Root exception is java.net.Conne ctException:
localhost:7001: Destination unreachable; nested exception is : java.net.ConnectException:
Connection refused: connect; No available rou ter to destination]