We Keep Coding

ksql-server refuses to boot up

I am facing the following problem using Confluent Open Source platform, version 4.1.0:
[2018-05-01 03:43:33,433] ERROR Failed to initialize TopicClient: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. (io.confluent.ksql.util.KafkaTopicClient:257)
Exception in thread "main" io.confluent.ksql.util.KsqlException: Could not fetch broker information. KSQL cannot initialize AdminClient.
at io.confluent.ksql.util.KafkaTopicClientImpl.init(KafkaTopicClientImpl.java:258)
at io.confluent.ksql.util.KafkaTopicClientImpl.<init>(KafkaTopicClientImpl.java:62)
at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:237)
at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:58)
at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:39)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
Changing the listener port didn't help. How do we fix this?
EDIT1: I am starting the kafka brokers and ksql-server using
confluent start
Initially, "confluent status" shows that the ksql-server is UP, but the server goes down after the above timeout.
EDIT2: Yes, my kafka broker is running and here is my kafka server.properties:
broker.id=100
listeners=PLAINTEXT://localhost:19090
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs-100
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
confluent.support.customer.id=anonymous
group.initial.rebalance.delay.ms=0
and ksql-server.properties:
bootstrap.servers=localhost:19090
listeners=http://localhost:18088
ksql.server.ui.enabled=true
EDIT3: My suspicion is this has got something to do with incorrect bootstrap server url, but I am not able to find that yet.
EDIT4: KSQL server logs, as requested.
[2018-05-17 03:41:33,244] INFO KsqlRestConfig values:
metric.reporters = []
ssl.client.auth = false
ksql.server.install.dir = /home/<user name>/confluent/confluent-4.1.0
response.mediatype.default = application/json
authentication.realm =
ssl.keystore.type = JKS
ssl.trustmanager.algorithm =
authentication.method = NONE
metrics.jmx.prefix = rest-utils
request.logger.name = io.confluent.rest-utils.requests
ssl.key.password = [hidden]
ssl.truststore.password = [hidden]
authentication.roles = [*]
metrics.num.samples = 2
ssl.endpoint.identification.algorithm =
compression.enable = false
query.stream.disconnect.check = 1000
ssl.protocol = TLS
debug = false
listeners = [http://localhost:18088]
ssl.provider =
ssl.enabled.protocols = []
shutdown.graceful.ms = 1000
ssl.keystore.location =
response.mediatype.preferred = [application/json]
ssl.cipher.suites = []
authentication.skip.paths = []
ssl.truststore.type = JKS
access.control.allow.methods =
access.control.allow.origin =
ssl.truststore.location =
ksql.server.command.response.timeout.ms = 5000
ssl.keystore.password = [hidden]
ssl.keymanager.algorithm =
port = 8080
metrics.sample.window.ms = 30000
metrics.tag.map = {}
ksql.server.ui.enabled = true
(io.confluent.ksql.rest.server.KsqlRestConfig:179)
[2018-05-17 03:41:33,302] INFO KsqlConfig values:
ksql.persistent.prefix = query_
ksql.schema.registry.url = http://localhost:8081
ksql.service.id = default_
ksql.sink.partitions = 4
ksql.sink.replicas = 1
ksql.sink.window.change.log.additional.retention = 1000000
ksql.statestore.suffix = _ksql_statestore
ksql.transient.prefix = transient_
(io.confluent.ksql.util.KsqlConfig:279)
[2018-05-17 03:43:33,433] ERROR Failed to initialize TopicClient: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. (io.confluent.ksql.util.KafkaTopicClient:257)
Exception in thread "main" io.confluent.ksql.util.KsqlException: Could not fetch broker information. KSQL cannot initialize AdminClient.
at io.confluent.ksql.util.KafkaTopicClientImpl.init(KafkaTopicClientImpl.java:258)
at io.confluent.ksql.util.KafkaTopicClientImpl.<init>(KafkaTopicClientImpl.java:62)
at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:237)
at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:58)
at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:39)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:258)
at io.confluent.ksql.util.KafkaTopicClientImpl.init(KafkaTopicClientImpl.java:230)
... 4 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.

I had this same exact problem on Confluent platform 5.4.1 and 5.3.1. I'm running MacOs 10.14.6. It turned out another application had taken port 8081 and therefore schema-registry was not able to bind it. I configured schema-registry to use port 8881 and reconfigured the schema-registry port on ksql-server configuration to that same value. This solved the problem
Therefore I would suggest you check that schema-registry is able to bind the configured port and that ksql-server is able to connect to that same port.

Spark streaming on Yarn Error while creating FlumeDStream java.net.BindException: Cannot assign requested address

I am trying to create spark stream from flume push based approach .I am running spark on my Yarn cluster.while starting the stream it is unable to bind the requested address.
I am using scala-shell to execute the program ,below is the code I am using
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.Seconds
import org.apache.spark.streaming.flume._
var ssc = new StreamingContext(sc,Seconds(60))
var stream = FlumeUtils.createStream(ssc,"master.internal", 5858);
stream.print()
stream.count().map(cnt => "Received " + cnt + " flume events." ).print()
ssc.start()
ssc.awaitTermination()
Flume Agent is unable to write to this port since this code is unable to bind 5858 port.
Flume Stack Trace :
[18-Dec-2014 15:20:13] [WARN] [org.apache.flume.sink.AbstractRpcSink.start(AbstractRpcSink.java:294) 294] Unable to create Rpc client using hostname: hostname, port: 5858
org.apache.flume.FlumeException: NettyAvroRpcClient { host: hadoop-master.nycloudlab.internal, port: 7575 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:178)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:118)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:624)
Caused by: java.io.IOException: Error connecting to /hostname:port
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:280)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:206)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:155)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:164)
... 18 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
... 3 more
Stack Trace from spark streaming as below.
14/12/18 19:57:48 ERROR scheduler.ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - org.jboss.netty.channel.ChannelException: Failed to bind to: <server-name>/IP:5858
at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:106)
at org.apache.spark.streaming.flume.FlumeReceiver.initServer(FlumeInputDStream.scala:157)
at org.apache.spark.streaming.flume.FlumeReceiver.onStart(FlumeInputDStream.scala:171)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257)
at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
at org.apache.spark.scheduler.Task.run(Task.scala:54)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.BindException: Cannot assign requested ad`enter code here`dress
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
... 3 more

This example (org.apache.spark.examples.streaming.FlumeEventCount) is worked:
// Create the context and set the batch size
val sparkConf = new SparkConf().setAppName("FlumeEventCount")
val ssc = new StreamingContext(sparkConf, batchInterval)
// Create a flume stream
val stream = FlumeUtils.createStream(ssc, host, port, StorageLevel.MEMORY_ONLY_SER_2)
// Print out the count of events received from this server in each batch
stream.count().map(cnt => "Received " + cnt + " flume events." ).print()
some hints:
use val instead of var
use exact ip instead of hostname or modify /etc/hosts in related nodes

Reading HDFS through java API

[possible duplicate]I am trying to read HDFS using java API. Using command line and Hadoop Url It's working fine
but problem is reading while hdfs paths.I went through this Reading HDFS and local files in Java but I am not able to find where i am wrong.
1)command line gives this result
hduser#hduser-Satellite:~$ hadoop fs -ls
Found 3 items
drwxr-xr-x - hduser supergroup 0 2014-01-11 00:21 /user/hduser/In
-rw-r--r-- 1 hduser supergroup 37461150 2014-01-11 17:27 /user/hduser/loging.txt
-rw-r--r-- 3 hduser supergroup 112383446 2014-01-11 19:02 /user/hduser/loging1.txt
2)
static {
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
}
InputStream in = null;
try {
*********************This is working fine********************************
in = new URL("hdfs://localhost:54310/user/hduser/loging.txt")
.openStream();
*************************************************************************
*************************This is not working*****************************
in = new URL("hdfs://localhost/user/hduser/loging.txt").openStream();
It says:
14/01/11 19:54:55 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:8020. Already tried 0 time(s).
.
.
14/01/11 19:55:04 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:8020. Already tried 9 time(s).
Exception in thread "main" java.net.ConnectException: Call to
localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused
**************************************************************************
IOUtils.copyBytes(in, System.out, 4096, false);
} finally {
IOUtils.closeStream(in);
}
3)Giving Exception
Configuration configuration = new Configuration();
configuration.addResource(new Path("/hadoop/conf/core-site.xml"));
configuration.addResource(new Path("/hadoop/conf/hdfs-site.xml"));
FileSystem fileSystem = FileSystem.get(configuration);
System.out.println(fileSystem.getHomeDirectory());
Path path = new Path("/user/hduser/loging.txt");
FSDataInputStream in = fileSystem.open(path);
System.out.println(in);
byte[] b = new byte[1024];
int numBytes = 0;
while ((numBytes = in.read(b)) > 0) {
//processing
}
in.close();
fileSystem.close();
Exception:
file:/home/hduser
Exception in thread "main" java.io.FileNotFoundException: File /user/hduser/loging.txt does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
at HdfsUrl.read(HdfsUrl.java:29)
at HdfsUrl.main(HdfsUrl.java:58)

The kind of exception you are getting suggests thats your application is trying to connect to a datanode that does not have 8020 port open. As per this documentation 8020 is the default port namenode. I'd suggest adding your hostname and port info. in your core-site.xml. something like this:
<property>
<name>fs.default.name</name>
<value>hdfs://[namenode]:[port]</value>
</property>

Sonar stops working every few days: jdbc connection error

I have a Sonar server which is used once a day from maven/jenkins and every few days, say every 4 or 5 days, it crashes and and shows the message "We're sorry, but something went wrong".
In the log, the error is always about a jdbc connection problem. I thought that it was a problem with the database but then if I restart the sonar server everything goes fine again.
So, it looks like a memory leak or something in the sonar server that makes it crash every few days until someone restarts it. Does that makes sense?. This is the configuration I have:
sonar.jdbc.username: xxxx
sonar.jdbc.password: xxxx
sonar.jdbc.url: jdbc:mysql://x.x.x.x:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true
#----- Connection pool settings
sonar.jdbc.maxActive: 20
sonar.jdbc.maxIdle: 5
sonar.jdbc.minIdle: 2
sonar.jdbc.maxWait: 5000
sonar.jdbc.minEvictableIdleTimeMillis: 600000
sonar.jdbc.timeBetweenEvictionRunsMillis: 30000
sonar.updatecenter.activate=true
http.proxyHost=xxxx
http.proxyPort=3128
sonar.notifications.delay=60
That's it. And this is the error log:
INFO o.s.s.p.ServerImpl SonarQube Server / 3.7.3 /
INFO o.s.c.p.Database Create JDBC datasource for jdbc:mysql://x.x.x.x:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true
ERROR o.s.c.p.Database Can not connect to database.
Please check connectivity and settings (see the properties prefixed by 'sonar.jdbc.').
org.apache.commons.dbcp.SQLNestedException:
Cannot create PoolableConnectionFactory (Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
.
.
.
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
.
.
.
Caused by: java.net.ConnectException: Connection refused
.
.
.
INFO jruby.rack An exception happened during JRuby-Rack startup
no connection available
--- System
jruby 1.6.8 (ruby-1.8.7-p357) (2012-09-18 1772b40) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_43) [linux-amd64-java]
Time: Thu Jan 02 08:04:08 -0500 2014
Server: jetty/7.6.11.v20130520
jruby.home: file:/opt/sonar/war/sonar-server/WEB-INF/lib/jruby-complete-1.6.8.jar!/META-INF/jruby.home
--- Context Init Parameters:
jruby.compat.version = 1.8
jruby.max.runtimes = 1
jruby.min.runtimes = 1
jruby.rack.logging = slf4j
public.root = /
rails.env = production
--- Backtrace
ActiveRecord::ConnectionNotEstablished: no connection available
set_native_database_types at arjdbc/jdbc/RubyJdbcConnection.java:517
initialize at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-jdbc-adapter-1.1.3/lib/arjdbc/jdbc/connection.rb:61
initialize at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-jdbc-adapter-1.1.3/lib/arjdbc/jdbc/adapter.rb:31
jdbc_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-jdbc-adapter-1.1.3/lib/arjdbc/jdbc/connection_methods.rb:6
send at org/jruby/RubyKernel.java:2109
new_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:223
checkout_new_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:245
checkout at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:188
loop at org/jruby/RubyKernel.java:1439
checkout at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:184
mon_synchronize at file:/opt/sonar/war/sonar-server/WEB-INF/lib/jruby-complete-1.6.8.jar!/META-INF/jruby.home/lib/ruby/1.8/monitor.rb:191
checkout at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:183
connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:98
retrieve_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:326
retrieve_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_specification.rb:123
connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_specification.rb:115
initialize at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/migration.rb:440
up at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/migration.rb:401
migrate at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/migration.rb:383
upgrade_and_start at /opt/sonar/war/sonar-server/WEB-INF/lib/database_version.rb:62
automatic_setup at /opt/sonar/war/sonar-server/WEB-INF/lib/database_version.rb:74
(root) at /opt/sonar/war/sonar-server/WEB-INF/config/environment.rb:213
load at org/jruby/RubyKernel.java:1087
load_environment at /opt/sonar/war/sonar-server/WEB-INF/config/environment.rb:23
load_environment at file:/opt/sonar/war/sonar-server/WEB-INF/lib/jruby-rack-1.1.10.jar!/jruby/rack/rails_booter.rb:65
(root) at <script>:1
--- RubyGems
Gem.dir: /opt/sonar/war/sonar-server/WEB-INF/gems
Gem.path:
/opt/sonar/war/sonar-server/WEB-INF/gems
Activated gems:
rake-0.9.2.2
activesupport-2.3.15
activerecord-2.3.15
rack-1.1.3
actionpack-2.3.15
actionmailer-2.3.15
activeresource-2.3.15
rails-2.3.15
color-tools-1.3.0
i18n-0.4.2
json-jruby-1.2.3-universal-java-1.6
activerecord-jdbc-adapter-1.1.3
fastercsv-1.4.0
--- Bundler
undefined method `bundle_path' for Bundler:Module
--- JRuby-Rack Config
compat_version = RUBY1_8
default_logger = org.jruby.rack.logging.StandardOutLogger#4fbbe4e1
err = java.io.PrintStream#d2284af
filter_adds_html = true
filter_verifies_resource = false
ignore_environment = false
initial_memory_buffer_size =
initial_runtimes = 1
jms_connection_factory =
jms_jndi_properties =
logger = org.jruby.rack.logging.Slf4jLogger#566dc8f0
logger_class_name = slf4j
logger_name = jruby.rack
maximum_memory_buffer_size =
maximum_runtimes = 1
num_initializer_threads =
out = java.io.PrintStream#6aeeefcf
rackup =
rackup_path =
rewindable = true
runtime_arguments =
runtime_timeout_seconds =
serial_initialization = false
servlet_context = ServletContext#o.e.j.w.WebAppContext{/,file:/opt/sonar/war/sonar-server/},file:/opt/sonar/war/sonar-server
ERROR jruby.rack unable to create shared application instance
org.jruby.rack.RackInitializationException: no connection available
.
.
.
org.jruby.exceptions.RaiseException:
(ConnectionNotEstablished) no connection available
.
.
.
ERROR jruby.rack Error: application initialization failed
org.jruby.rack.RackInitializationException: no connection available
.
.
.
org.jruby.exceptions.RaiseException:
(ConnectionNotEstablished) no connection available
Any help will be appreciated :)

This sounds familiar to me :P Try asking your operators if they have some automated 'cleansing' operation that just kills periodically open database connections automatically in order to prevent leaked connections to the database.

It happened to me, in a Windows 2012 Server and SQL Server 2012 database in a different server. It seems that sonar service maintains a connection open, created at start up time, so any disconnection (networking, database restart, etc.) causes this unrecoverable connection problem. Restarting the Sonar (SonarQube) Windows service solved the problem to me. But, if this problem is frequent, as in your case, it would be a good idea to schedule a service restart task, or find out what is taking the connection down.

Java NIO Selector can select no more than 50 SelectionKeys?

I use siege to stress test my hand built file server, it works pretty well for small files(less than 1KB), while when tested with a 1MB file, it does not work as expected.
The following is the result of the test with a small file:
neevek#~$ siege -c 1000 -r 10 -b http://127.0.0.1:9090/1KB.txt
** SIEGE 2.71
** Preparing 1000 concurrent users for battle.
The server is now under siege.. done.
Transactions: 10000 hits
Availability: 100.00 %
Elapsed time: 9.17 secs
Data transferred: 3.93 MB
Response time: 0.01 secs
Transaction rate: 1090.51 trans/sec
Throughput: 0.43 MB/sec
Concurrency: 7.29
Successful transactions: 10000
Failed transactions: 0
Longest transaction: 1.17
Shortest transaction: 0.00
The following is the result of a test with a 1MB file:
neevek#~$ siege -c 1000 -r 10 -b http://127.0.0.1:9090/1MB.txt
** SIEGE 2.71
** Preparing 1000 concurrent users for battle.
The server is now under siege...[error] socket: read error Connection reset by peer sock.c:460: Connection reset by peer
[error] socket: unable to connect sock.c:222: Connection reset by peer
[error] socket: unable to connect sock.c:222: Connection reset by peer
[error] socket: unable to connect sock.c:222: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:460: Connection reset by peer
[error] socket: unable to connect sock.c:222: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:460: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:460: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:460: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:460: Connection reset by peer
When siege terminates with the above errors, my file server still spins with a fixed number of WRITABLE SelectionKey, i.e. Selector.select() keeps returning a fixed number, say 50.
With the above tests, it looks to me that my file server cannot accept no more than 50 concurrent connections, because when running the test with small file, I notice that the server selects 1 or 2 SelectionKeys, when running with big file, it selects up to 50 every time.
I tried to increase backlog in Socket.bind() with no help.
What could be the cause of the problem?
EDIT
More info:
When testing with a 1MB file, I noticed that siege terminated with a Broken pipe error, and the file server only accepted 198 connections, though I specified 1000 concurrent connections x 10 rounds(1000*10=10000) to flood the server.
EDIT 2
I have tested with the following code(a single class) to reproduce the same problem, in this code, I only accept connections, I don't read or write, the siege client terminated with Connection reset or Broken pipe error before connections time out. I also noticed that Selector can only select less than 1000 keys. you may try the code below to witness the problem.
public class TestNIO implements Runnable {
ServerSocketChannel mServerSocketChannel;
Selector mSelector;
public static void main(String[] args) throws Exception {
new TestNIO().start();
}
public TestNIO () throws Exception {
mSelector = Selector.open();
}
public void start () throws Exception {
mServerSocketChannel = ServerSocketChannel.open();
mServerSocketChannel.configureBlocking(false);
mServerSocketChannel.socket().bind(new InetSocketAddress(9090));
mServerSocketChannel.socket().setSoTimeout(150000);
mServerSocketChannel.register(mSelector, SelectionKey.OP_ACCEPT);
int port = mServerSocketChannel.socket().getLocalPort();
String serverName = "http://" + InetAddress.getLocalHost().getHostName() + ":" + port;
System.out.println("Server start listening on " + serverName);
new Thread(this).start();
}
#Override
public void run() {
try {
Thread.currentThread().setPriority(Thread.MIN_PRIORITY);
while (true) {
int num = mSelector.select();
System.out.println("SELECT = " + num + "/" + mSelector.keys().size());
if (num > 0) {
Iterator<SelectionKey> keys = mSelector.selectedKeys().iterator();
while (keys.hasNext()) {
final SelectionKey key = keys.next();
if (key.isValid() && key.isAcceptable()) {
accept(key);
}
}
// clear the selected keys
mSelector.selectedKeys().clear();
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
private void accept (SelectionKey key) throws IOException {
SocketChannel socketChannel = mServerSocketChannel.accept();
socketChannel.configureBlocking(false);
socketChannel.socket().setSoTimeout(1000000);
socketChannel.socket().setKeepAlive(true);
// since we are connected, we are ready to READ
socketChannel.register(mSelector, SelectionKey.OP_READ);
}
}

It is actually related the the default backlog value set for the ServerSocketChannel
http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/sun/nio/ch/ServerSocketChannelImpl.java#138
You can fix the issue by passing the backlog value as a second parameter to the bind method.
mServerSocketChannel.socket().bind(new InetSocketAddress(9090), "backlog value")

Check the ulimit and hard limit of the number of open files (file descriptors)
I'm guessing you're using linux. You can look in limits.conf
/etc/security/limits.conf

This problem may not relate to my code, I run the same test against an nginx server running locally(MacOSX), the same error occurred. So it most likely relates to hardware or the siege client.