A code that worked me for a while now on various platforms started to suddenly to fail, there were changed in the code but not in an area that related to this place.
The line of code that throws the exceptions is:
(SocketChannel) selectionKey.channel()).finishConnect()
And it throws:
java.net.SocketException: Permission denied: no further information
what may be the issue?
Some log lines that might show more info, it is a test code:
2016-12-04 22:50:10,585 [main] DEBUG Socket - Socket-Client-1-1
register to connect to address: 0.0.0.0/0.0.0.0:8080
2016-12-04 22:50:10,585 [main] DEBUG SocketSelector - Registering keys:
OP_CONNECT (for Socket-Client-1-1)
2016-12-04 22:50:10,595 [main] DEBUG SocketSelector - Registered to: OP_CONNECT > 2016-12-04 22:50:10,595 [NetworkThread] DEBUG SocketSelector -Got selected keys
for channel (java.nio.channels.SocketChannel[connection-pending
remote=0.0.0.0/0.0.0.0:8080])
The issue is that you are attempting to connect to 0.0.0.0, which is not a valid TCP target address. This causes EPERM for the reasons described in man connect.
Use 127.0.0.1 or a proper IP address.
Related
I have a Netty application that keeps track of connected web socket connections. When a web socket disconnects, I need to perform some logic that cleans up the stored data on that connection. I'm trying to be a good Netty citizen and not block the event loop with this cleanup logic, so I'm running everything in a CompletableFuture. The logic looks vaguely like this:
#Override
public void channelInactive(ChannelHandlerContext ctx) {
String connectionId = ctx.channel().id().asLongText();
LOGGER.info("Web socket disconnected: {}.", connectionId);
connectionStore
.deleteConnection(connectionId)
.thenCompose(
(connection) → {
LOGGER.info("Deleted connection {}, publishing event.", connectionId);
return messageBusPublisher.publish(new DisconnectEvent(connectionId));
})
.thenAccept((v) -> LOGGER.info("Done cleaning up connection {}.", connectionId));
}
This results in log statements like this:
2020-02-10 17:47:23,091 [nioEventLoopGroup-6-2] INFO c.e.WebSocketFrameHandler - Web socket disconnected: acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2.
2020-02-10 17:47:23,091 [nioEventLoopGroup-6-2] INFO c.e.store.ConnectionStore - Attempting to delete connection for acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2
2020-02-10 17:47:23,092 [nioEventLoopGroup-6-1] INFO c.e.WebSocketFrameHandler - Web socket disconnected: acde48fffe001122-000080ec-00000003-60f2c3402164ad79-a7e03a19.
2020-02-10 17:47:23,092 [nioEventLoopGroup-6-1] INFO c.e.store.ConnectionStore - Attempting to delete connection for acde48fffe001122-000080ec-00000003-60f2c3402164ad79-a7e03a19
2020-02-10 17:47:23,093 [nioEventLoopGroup-6-3] INFO c.e.WebSocketFrameHandler - Web socket disconnected: acde48fffe001122-000080ec-00000005-6962838a219b52e9-e6ceac8b.
2020-02-10 17:47:23,093 [nioEventLoopGroup-6-3] INFO c.e.store.ConnectionStore - Attempting to delete connection for acde48fffe001122-000080ec-00000005-6962838a219b52e9-e6ceac8b
2020-02-10 17:47:23,093 [nioEventLoopGroup-6-5] INFO c.e.WebSocketFrameHandler - Web socket disconnected: acde48fffe001122-000080ec-00000007-ccfcc4a5219b57eb-d78fc5f5.
2020-02-10 17:47:23,093 [nioEventLoopGroup-6-4] INFO c.e.WebSocketFrameHandler - Web socket disconnected: acde48fffe001122-000080ec-00000006-6c933ab1219b554f-2b3cf759.
2020-02-10 17:47:23,093 [nioEventLoopGroup-6-5] INFO c.e.store.ConnectionStore - Attempting to delete connection for acde48fffe001122-000080ec-00000007-ccfcc4a5219b57eb-d78fc5f5
2020-02-10 17:47:23,093 [nioEventLoopGroup-6-4] INFO c.e.store.ConnectionStore - Attempting to delete connection for acde48fffe001122-000080ec-00000006-6c933ab1219b554f-2b3cf759
2020-02-10 17:47:23,097 [lettuce-kqueueEventLoop-4-1] INFO c.e.store.ConnectionStore - Successfully deleted connection for acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2
2020-02-10 17:47:23,102 [lettuce-kqueueEventLoop-4-1] INFO c.e.WebSocketFrameHandler - Deleted connection acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2, publishing event.
2020-02-10 17:47:23,127 [lettuce-kqueueEventLoop-4-1] INFO c.e.service.MessageBusPublisher - Attempting to publish message for acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2
2020-02-10 17:47:23,866 [sdk-async-response-0-0] INFO c.e.service.MessageBusPublisher - Successfully published message for acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2
2020-02-10 17:47:23,866 [sdk-async-response-0-0] INFO c.e.WebSocketFrameHandler - Done cleaning up connection acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2
The problem I'm running into is that when multiple disconnects happen in short order, it seems that the full cleanup logic is only run for the first one or two connections, and beyond that, nothing after the initial call is processed. This means the connections are properly deleted from the store, but no follow up actions are processed beyond that. My assumption is that this is because the CompletableFuture that represents the full chain of calls is not bound to anything, so when the first calls in the chain complete, the callbacks are not executed.
I've tried various things like using ctx.channel().eventLoop().execute(...) to run the clean up code (and theoretically bind the execution to the channel's event loop), but I see the same results.
My question then is: what is the appropriate way to fire off I/O bound tasks in response to Netty channel events?
As discussed above in the comments, this had nothing to do w/ how I was handling the long running work, but instead was the lack of error handling on my futures.
Remember to always catch exceptions from your futures, kids!
I changed the MQIVP sample in MQ with a server connection channel of my own local.server.con and it is working fine. But I tried connecting to the same channel with PCFMessageAgent and the connection is failing with errors in MQ log. What is the relation between my channel and SYSTEM.DEFAULT.MODEL.QUEUE which gives the error.
C:\Program Files\IBM\WebSphere MQ\Tools\wmqjava\samples>java -Djava.library.path="C:\Program Files\IBM\WebSphere MQ\java\lib" MQIVPMod
Websphere MQ for Java Installation Verification Program
5724-B4 (C) Copyright IBM Corp. 2002, 2014. All Rights Reserved.
================================================================
Please enter the IP address of the MQ server :10.40.1.16
Please enter the port to connect to : (1414)1415
Please enter the server connection channel name :local.server.con
Please enter the user name (or RETURN for none) :test
Please enter the password for the user :test123
Please enter the queue manager name :local
Success: Connected to queue manager.
Success: Opened SYSTEM.DEFAULT.LOCAL.QUEUE
Success: Put a message to SYSTEM.DEFAULT.LOCAL.QUEUE
Success: Got a message from SYSTEM.DEFAULT.LOCAL.QUEUE
Success: Closed SYSTEM.DEFAULT.LOCAL.QUEUE
Success: Disconnected from queue manager
Tests complete -
SUCCESS: This MQ Transport is functioning correctly.
Press Enter to continue ...
My PCFMessageAgent code and error:
new PCFMessageAgent(host, Integer.parseInt(port), channelName); // connect
com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2035'.
at com.ibm.mq.MQDestination.open(MQDestination.java:323)
at com.ibm.mq.MQQueue.<init>(MQQueue.java:236)
at com.ibm.mq.MQQueueManager.accessQueue(MQQueueManager.java:2674)
at com.ibm.mq.pcf.PCFAgent.open(PCFAgent.java:448)
at com.ibm.mq.pcf.PCFAgent.open(PCFAgent.java:394)
at com.ibm.mq.pcf.PCFAgent.connect(PCFAgent.java:287)
at com.ibm.mq.pcf.PCFAgent.<init>(PCFAgent.java:190)
at com.ibm.mq.pcf.PCFMessageAgent.<init>(PCFMessageAgent.java:157)
at test.wmq.PCFTest.main(PCFTest.java:49)
And the MQ log :
5/2/2017 14:01:31 - Process(6048.60) User(MUSR_MQADMIN) Program(amqzlaa0.exe)
Host(BLR_SWG_N09505) Installation(Installation1)
VRMF(8.0.0.4) QMgr(local)
AMQ8077: Entity 'test#blr_swg_n09505' has insufficient authority to access
object 'SYSTEM.DEFAULT.MODEL.QUEUE'.
EXPLANATION:
The specified entity is not authorized to access the required object. The
following requested permissions are unauthorized: get
ACTION:
Ensure that the correct level of authority has been set for this entity against
the required object, or ensure that the entity is a member of a privileged
group.
----- amqzfubn.c : 518 --------------------------------------------------------
5/2/2017 14:01:32 - Process(8004.41) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(BLR_SWG_N09505) Installation(Installation1)
VRMF(8.0.0.4) QMgr(local)
AMQ9208: Error on receive from host BLR_SWG_N09505 (10.40.1.16).
EXPLANATION:
An error occurred receiving data from BLR_SWG_N09505 (10.40.1.16) over TCP/IP.
This may be due to a communications failure.
ACTION:
The return code from the TCP/IP recv() call was 10054 (X'2746'). Record these
values and tell the systems administrator.
----- amqccita.c : 4076 -------------------------------------------------------
5/2/2017 14:01:32 - Process(8004.41) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(BLR_SWG_N09505) Installation(Installation1)
VRMF(8.0.0.4) QMgr(local)
AMQ9999: Channel 'local.server.con' to host '10.40.1.16' ended abnormally.
EXPLANATION:
The channel program running under process ID 8004(7988) for channel
'local.server.con' ended abnormally. The host name is '10.40.1.16'; in some
cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
----- amqrmrsa.c : 930 --------------------------------------------------------
You need to go and read up on MQ permissions (i.e. authorizations). It is best to do permissions on the group rather than principle (UserId).
setmqaut -m {QM_NAME} -n SYSTEM.ADMIN.COMMAND.QUEUE -t queue -g {GROUP} +put +inq +dsp
setmqaut -m {QM_NAME} -n SYSTEM.DEFAULT.MODEL.QUEUE -t queue -g {GROUP} +get +inq +dsp
There is no relation between channels and model queues.
But I think, that the PCFMessageAgent is trying to create a dynamic queue to use as a ReplyToQ to receive responses, and it seems it tries to create the dynamic queue by opening the SYSTEM.DEFAULT.MODEL.QUEUE.
I have a ThreadPoolExecutorService to which I'm submitting runnable jobs that are uploading large (1-2 GB) files to Amazon's S3 file system, using the AWS Java SDK. Occasionally one of my worker threads will report a java.net.SocketException with "Connection reset" as the cause and then die.
AWS doesn't use checked exceptions so I actually can't catch SocketException directly---it must be wrapped somehow. My question is how I should deal with this problem so I can retry any problematic uploads and increase the reliability of my program.
Would the Multipart Upload API be more reliable?
Is there some exception I can reliably catch to enable retries?
Here's the stack trace. The com.example.* code is mine. Basically what the DataProcessorAWS call does is call putObject(String bucketName, String key, File file) on an instance of AmazonS3Client that's shared across threads.
14/12/11 18:43:17 INFO http.AmazonHttpClient: Unable to execute HTTP request: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:377)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:363)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:830)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:169)
at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:119)
at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:102)
at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:153)
at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197)
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:47)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:685)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3697)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1434)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1294)
at com.example.DataProcessorAWS$HitWriter.close(DataProcessorAWS.java:156)
at com.example.DataProcessorAWS$Processor.run(DataProcessorAWS.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
For that you have to use only AmazonS3Client not Trasfermanager for upload and download
U have to configure AmazonS3Client with following properties:
1. connectionTimeout=50000 in ms
2.maxConnections=500
3.socketTimeout=50000 in ms
4.maxErrorRetry=10
For upload use
AmazonS3Client.putObject(bucketName, key, inputFile);
For download use
S3Object s3Object = AmazonS3Client.getObject(new GetObjectRequest(bucketName, key));`
InputStream downloadStream = s3Object.getObjectContent();
and store that stream by reading bytes.
'Connection reset' means the connection is closed. Close the socket, or whatever higher-level construct you're using. Probably the server has decided the upload is too large, or it's overloaded, or something. Whether you can retry the operation is something only you can know.
In my case, I am saving a object with some metadata. During the prototype, we added a system metadata "time-created", and that consistently resulted into the similar error as above in the question. And it is not retry-able as it intends to modify a system metadata. But according to the API, as below,
Throws:
SdkClientException - If any errors are encountered in the client
while making > the request or handling the response.
AmazonServiceException - If any errors occurred in Amazon S3 while processing > the request.
My understanding is that the expected behavior should be,
throw one of the above exceptions, OR,
return an 403, rather than an umbrella AmazonClientException, "Connection shutdown".
See the logs below when S3 debug is turned on,
2017-07-11 17:12:45,163 DEBUG [org.apache.http.wire] - http-outgoing-3 >> "**[write] I/O error: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset"**
2017-07-11 17:12:45,221 DEBUG [org.apache.http.wire] - http-outgoing-4 >> "PUT /dev-ec-dev/neon/ECTE/CPER/2017-6/ECTE_dp0p_CPER_2017-06-05.h5 HTTP/1.1[\r][\n]"
2017-07-11 17:12:45,221 DEBUG [org.apache.http.wire] - http-outgoing-4 >> "Host: s3.data.neonscience.org[\r][\n]"
2017-07-11 17:12:45,221 DEBUG [org.apache.http.wire] - http-outgoing-4 >> "Authorization: AWS dev-ec-owner:dE6ouKtPpP9aftLRba4OVn6DH9M=[\r][\n]"
2017-07-11 17:12:45,222 DEBUG [org.apache.http.wire] - http-outgoing-4 >> "x-amz-meta-site: CPER[\r][\n]"
2017-07-11 17:12:45,226 DEBUG [org.apache.http.wire] - http-outgoing-4 >> "x-amz-meta-time-created: 2017-07-11T17:12:44.904[\r][\n]"
More logs with regular logging,
2017-07-11 17:12:45,247 DEBUG [org.apache.http.wire] - http-outgoing-4 >> "[write] I/O error: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset"
2017-07-11 17:12:45,248 ERROR [org.battelle.neon.is.transition.ec.ECTELevelZeroPrimeJob] - Error copyDataToS3: Not getting descriptive AmazonS3Exception.
Batch runner failed.com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection reset
java.lang.RuntimeException: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection reset
at org.battelle.neon.is.transition.ec.ECTELevelZeroPrimeJob.runTransition(ECTELevelZeroPrimeJob.java:347)
at org.battelle.neon.is.batch.BatchRunner.lambda$null$1(BatchRunner.java:152)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
The javadoc says that putObject() throws AmazonClientException when this kind of error occurs. AmazonClientException has a method called isRetryable(), you can try with that.
I have the following problem:
I try to connect to an ActiveMQ broker (which is now down) using the following piece of code
connectionFactory = new ActiveMQConnectionFactory(this.url + "?timeout=2000");
connection = connectionFactory.createConnection();
connection.start();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
LOGGER.info("Connected to " + this.url);
The problem is that the timeout does not have any effect
connection.start()
is blocked forever.
I inspected ActiveMQ log and found the following info:
2013-12-20 01:49:03,149 DEBUG [ActiveMQ Task-1] (FailoverTransport.java:786) - urlList connectionList:[tcp://localhost:61616?timeout=2000], from: [tcp://localhost:61616?timeout=2000]
2013-12-20 01:49:03,149 DEBUG [ActiveMQ Task-1] (FailoverTransport.java:1040) - Connect fail to: tcp://localhost:61616?timeout=2000, reason: java.lang.IllegalArgumentException: Invalid connect parameters: {timeout=2000}
The timeout parameter is specified here http://activemq.apache.org/cms/configuring.html
Has anybody any idea how to pass timeout argument to ActiveMQConnectionFactory?
Or how to set a timeout for connection.start() ?
Thank you!
Update: I found this on Stackoverflow: ActiveMQ - CreateSession failover timeout after a connection is resumed . I tried it but the following exception is thrown:
javax.jms.JMSException: Could not create Transport. Reason: java.lang.IllegalArgumentException: Invalid connect parameters: {transport.timeout=5000}
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
I use ActiveMQ 5.8.0 from maven repo
It appears that your url is invalid still in both cases when attempting to set the timeout property.
If you're trying to have a failover URL, which it looks like you are since it is getting in to the Failover code then you're probably looking for initialReconnectDelay (and possibly maxReconnectAttempts which would throw an exception if the server is still down after the number of attempts is reached).
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("failover://(tcp://localhost:61616)?initialReconnectDelay=2000&maxReconnectAttempts=2");
I'm trying to connect to remote HDFS cluster. I've read some documentation and getting started's but didn't find a best solution how to do that.
Situation: I have HDFS on xxx-something.com. I can connect to it via SSH and everything works.
But what I'm trying to do, get the files from it to my local machine.
What I've done:
I've created core-site.xml in my conf folder (I'm creating Play! application). There I've changed fs.default.name config to hdfs://xxx-something.com:8020 (not sure about the port).
Then I'm trying to launch a simple test:
val conf = new Configuration()
conf.addResource(new Path("conf/core-site.xml"))
val fs = FileSystem.get(conf)
val status = fs.listStatus(new Path("/data/"))
And I'm getting errors:
13:56:09.012 [specs2.DefaultExecutionStrategy1] WARN org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: fs.trash.interval; Ignoring.
13:56:09.012 [specs2.DefaultExecutionStrategy1] WARN org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: hadoop.tmp.dir; Ignoring.
13:56:09.013 [specs2.DefaultExecutionStrategy1] WARN org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: fs.checkpoint.dir; Ignoring.
13:56:09.022 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.fs.FileSystem - Creating filesystem for hdfs://xxx-something.com:8021
13:56:09.059 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.conf.Configuration - java.io.IOException: config()
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:226)
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:213)
at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:53)
at org.apache.hadoop.net.NetUtils.<clinit>(NetUtils.java:62)
Thanks in advance!
Update:
probably the port was wrong. Now I set it to 22, I'm still getting same errors, but after 3 times it does say:
14:01:01.877 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.ipc.Client - Connecting to xxx-something.com/someIp:22
14:01:02.187 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to xxx-something.com/someIp:22 from britva sending #0
14:01:02.188 [IPC Client (47) connection to xxx-something.com/someIp:22 from britva] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to xxx-something.com/someIp:22 from britva: starting, having connections 1
14:01:02.422 [IPC Client (47) connection to xxx-something.com/someIp:22 from britva] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to xxx-something.com/someIp:22 from britva got value #1397966893
And afterwards:
Call to xxx-something.com/someIp:22 failed on local exception: java.io.EOFException
java.io.IOException: Call to xxx-something.com/someIp:22 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
at org.apache.hadoop.ipc.Client.call(Client.java:1071)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:122)
at HdfsSpec$$anonfun$1$$anonfun$apply$3.apply(HdfsSpec.scala:33)
at HdfsSpec$$anonfun$1$$anonfun$apply$3.apply(HdfsSpec.scala:17)
at testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.executeBody(MyNotifierRunner.scala:16)
at testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.execute(MyNotifierRunner.scala:16)
Caused by: java.io.EOFException
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:807)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:745)
What does it mean?
You'll need to find the fs.default.name property in the $HADOOP_HOME/conf/core-site.xml on the server running the Name Node (HDFS master) to get the correct port. It might be 8020, or it could be something else. That's what you should use. Make sure there's no firewall between you and the server that disallows connections on the port.