Our subscription client connections keeps disconnecting with the following error.
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:47)
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:36)
at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:37)
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:37)
at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:210)
at org.springframework.data.redis.connection.jedis.JedisConnection.subscribe(JedisConnection.java:2986)
at org.springframework.data.redis.listener.RedisMessageListenerContainer$SubscriptionTask.eventuallyPerformSubscription(RedisMessageListenerContainer.java:784)
at org.springframework.data.redis.listener.RedisMessageListenerContainer$SubscriptionTask.run(RedisMessageListenerContainer.java:740)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Connection reset
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:202)
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40)
at redis.clients.jedis.Protocol.process(Protocol.java:151)
at redis.clients.jedis.Protocol.read(Protocol.java:215)
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340)
at redis.clients.jedis.Connection.getRawObjectMultiBulkReply(Connection.java:285)
at redis.clients.jedis.BinaryJedisPubSub.process(BinaryJedisPubSub.java:87)
at redis.clients.jedis.BinaryJedisPubSub.proceed(BinaryJedisPubSub.java:82)
at redis.clients.jedis.BinaryJedis.subscribe(BinaryJedis.java:3070)
at org.springframework.data.redis.connection.jedis.JedisConnection.subscribe(JedisConnection.java:2983)
... 5 common frames omitted
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:127)
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:196)
... 14 common frames omitted
]
For having permanent connections, we have configured 0 timeout in both client and server.
Redis Client:
poolConfig.setMaxTotal(10);
poolConfig.setMaxIdle(5);
poolConfig.setBlockWhenExhausted(true);
poolConfig.setMinIdle(2);
poolConfig.setTestOnBorrow(true);
JedisConnectionFactory factory = new JedisConnectionFactory();
factory.setHostName(HOST);
factory.setPort(PORT);
factory.setDatabase(Protocol.DEFAULT_DATABASE);
factory.setPoolConfig(poolConfig);
factory.setTimeout(0);
JedisShardInfo shardInfo = new JedisShardInfo(factory.getHostName(), factory.getPort(), 0);
factory.setShardInfo(shardInfo);
return factory;
Redis Server Config:
127.0.0.1:6379> CONFIG GET client-output-buffer-limit
1) "client-output-buffer-limit"
2) "normal 0 0 0 slave 268435456 67108864 60 pubsub 257949696 134217728 0"
Can someone please help?
Related
I have an AWS MSK cluster up and running. Connected to it and ran this command to create a test topic called topicoteste
usr/local/kafka_2.13-2.5.0/bin/kafka-topics --create --bootstrap-server BOOTSTRAP_STRING_HERE --partitions 1 --replication-factor 3 --topic topicoteste
These are the two errors I get. Any suggestions?
Error while executing topic command : org.apache.kafka.common.errors.TimeoutException: Call(callName=listTopics, deadlineMs=1611587423888) timed out at 9223372036854775807 after 1 attempt(s)
[2021-01-25 15:09:24,312] ERROR Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1': (org.apache.kafka.common.utils.KafkaThread)
java.lang.OutOfMemoryError: Java heap space
at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:61)
at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:348)
at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:113)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:448)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:398)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:678)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:580)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1272)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1203)
at java.base/java.lang.Thread.run(Thread.java:829)
[2021-01-25 15:09:24,314] ERROR java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listTopics, deadlineMs=1611587423888) timed out at 9223372036854775807 after 1 attempt(s)
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:227)
at kafka.admin.TopicCommand$TopicService.createTopic(TopicCommand.scala:196)
at kafka.admin.TopicCommand$TopicService.createTopic$(TopicCommand.scala:191)
at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:219)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:62)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listTopics, deadlineMs=1611587423888) timed out at 9223372036854775807 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.
(kafka.admin.TopicCommand$)
I had the same problem because the broker used TLS and the AdminClient was not configured to use TLS.
You can either run a PLAINTEXT-listener next to the TLS listener and use that to create topics or configure your admin client with --command-config <ssl.conf> and a file ssl.conf looking something like this:
ssl.endpoint.identification.algorithm=https
security.protocol=SSL
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=password
ssl.key.password=password
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=password
I have a spring boot applicaiton running which connects to mongoserver in azure cloud.
The app continuesly gets java.net.SocketException: Connection reset exception for the first request after 4 secs idletime and fail.
Then the subsequent requests are processed after a new connection is established.
Mongo client configuration
#Bean
public MongoClient mongoClient() {
final String address = serverAddress;
MongoCredential credential = MongoCredential.createCredential(userName, database, primaryPassword.toCharArray());
ServerAddress sAddress = new ServerAddress(address, port);
MongoClientOptions.Builder opBuilder = MongoClientOptions.builder();
opBuilder.sslEnabled(true);
opBuilder.requiredReplicaSetName(requiredReplicaSetName);
opBuilder.maxConnectionIdleTime(1500000);
opBuilder.socketTimeout(30000);
opBuilder.connectTimeout(30000);
// MongoClient
return new MongoClient(sAddress, credential, opBuilder.build());
}
and the exception:
2020-08-05T11:40:55.34-0400 [APP/PROC/WEB/0] OUT Caused by: org.springframework.data.mongodb.UncategorizedMongoDbException: Exception sending message; nested exception is com.mongodb.MongoSocketWriteException: Exception sending message
2020-08-05T11:40:55.34-0400 [APP/PROC/WEB/0] OUT
at org.springframework.data.mongodb.repository.query.AbstractMongoQuery.lambda$getExecution$1(AbstractMongoQuery.java:115) ~[spring-data-mongodb-2.1.14.RELEASE.jar!/:2.1.14.RELEASE]
2020-08-05T11:40:55.34-0400 [APP/PROC/WEB/0] OUT at
... 98 common frames omitted
2020-08-05T11:40:55.34-0400 [APP/PROC/WEB/0] OUT Caused by: com.mongodb.MongoSocketWriteException: Exception sending message
2020-08-05T11:40:55.34-0400 [APP/PROC/WEB/0] OUT at com.mongodb.internal.connection.InternalStreamConnection.translateWriteException(InternalStreamConnection.java:525) ~[mongo-java-driver-3.8.2.jar!/:na]
2020-08-05T11:40:55.34-0400 [APP/PROC/WEB/0] OUT ... 130 common frames omitted
2020-08-05T11:40:55.34-0400 [APP/PROC/WEB/0] OUT Caused by: java.net.SocketException: Connection reset
2020-08-05T11:40:55.34-0400 [APP/PROC/WEB/0] OUT ... 152 common frames omitted
I tried changing the connection timeout, socketimeout and maxConnectionIdleTime to increase/decrese but the app continuesly gets this exception after 4sec of idle time.
I have been trying to connect to an RabbitMQ when the broker is in a shutdown state. So far I have:
com.rabbitmq.client.ConnectionFactory factory = new
com.rabbitmq.client.ConnectionFactory();
factory.setHost("localhost");
factory.setPort(5672);
factory.setAutomaticRecoveryEnabled(true);
factory.setNetworkRecoveryInterval(100000);
factory.setConnectionTimeout(100000);
factory.setHandshakeTimeout(100000);
return factory.newConnection();
However when factory.newConnection() is executed when the application startup. Here is the stacktrace:
Caused by: java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:124)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:120)
at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:362)
at com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:63)
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:99)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:918)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:877)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:835)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:987)
at com.app.RabbitMqConectionFactory.provide(RabbitMqConectionFactory.java:41)
... 75 more
Caused by: com.rabbitmq.client.ShutdownSignalException: connection error
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:443)
at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:306)
... 82 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:91)
at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:164)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:580)
at java.lang.Thread.run(Thread.java:745)
Using redisTemplate, things work until redis server restart. Then two exceptions occur (see below).
When I stop using redisTemplate.executePipelined, the exception 2 stops happening. After invoking redisTemplate.setEnableTransactionSupport(false);, the exception 1 goes away.
But I have no idea why.
Any ideas?
Exception 1:
org.springframework.data.redis.RedisConnectionFailureException: java.net.SocketException: Broken pipe; nested exception is redis.clients.jedis.ex
ceptions.JedisConnectionException: java.net.SocketException: Broken pipe
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:67)
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:41)
at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:37)
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:37)
at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:212)
at org.springframework.data.redis.connection.jedis.JedisConnection.hGetAll(JedisConnection.java:2890)
at org.springframework.data.redis.connection.DefaultStringRedisConnection.hGetAll(DefaultStringRedisConnection.java:360)
at org.springframework.data.redis.core.DefaultHashOperations$13.doInRedis(DefaultHashOperations.java:223)
at org.springframework.data.redis.core.DefaultHashOperations$13.doInRedis(DefaultHashOperations.java:220)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:204)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:166)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:88)
at org.springframework.data.redis.core.DefaultHashOperations.entries(DefaultHashOperations.java:220)
....
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Broken pipe
at redis.clients.jedis.Protocol.sendCommand(Protocol.java:104)
at redis.clients.jedis.Protocol.sendCommand(Protocol.java:84)
at redis.clients.jedis.Connection.sendCommand(Connection.java:127)
at redis.clients.jedis.BinaryClient.hgetAll(BinaryClient.java:294)
at redis.clients.jedis.BinaryJedis.hgetAll(BinaryJedis.java:1027)
at org.springframework.data.redis.connection.jedis.JedisConnection.hGetAll(JedisConnection.java:2888)
... 14 common frames omitted
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at redis.clients.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52)
at redis.clients.util.RedisOutputStream.write(RedisOutputStream.java:59)
at redis.clients.jedis.Protocol.sendCommand(Protocol.java:90)
... 19 common frames omitted
Exception 2:
org.springframework.dao.InvalidDataAccessApiUsageException: Cannot use Jedis when in Pipeline. Please use Pipeline or reset jedis state .; nested
exception is redis.clients.jedis.exceptions.JedisDataException: Cannot use Jedis when in Pipeline. Please use Pipeline or reset jedis state .
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:64)
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:41)
at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:37)
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:37)
at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:212)
at org.springframework.data.redis.connection.jedis.JedisConnection.hGetAll(JedisConnection.java:2890)
at org.springframework.data.redis.connection.DefaultStringRedisConnection.hGetAll(DefaultStringRedisConnection.java:360)
at org.springframework.data.redis.core.DefaultHashOperations$13.doInRedis(DefaultHashOperations.java:223)
at org.springframework.data.redis.core.DefaultHashOperations$13.doInRedis(DefaultHashOperations.java:220)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:204)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:166)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:88)
at org.springframework.data.redis.core.DefaultHashOperations.entries(DefaultHashOperations.java:220)
...
Caused by: redis.clients.jedis.exceptions.JedisDataException: Cannot use Jedis when in Pipeline. Please use Pipeline or reset jedis state .
at redis.clients.jedis.BinaryJedis.checkIsInMultiOrPipeline(BinaryJedis.java:1761)
at redis.clients.jedis.BinaryJedis.hgetAll(BinaryJedis.java:1026)
at org.springframework.data.redis.connection.jedis.JedisConnection.hGetAll(JedisConnection.java:2888)
... 14 common frames omitted
this works well:
scan = new Scan(startRow, stopRow);
this throws exception sometimes:
scan = new Scan(stopRow, startRow);
scan.setReversed(true);
throwing exception while traffic is 100 req/s. there are actually no timeouts, exception is fired immediately for 1-10% requests
hbase: 0.98.12-hadoop2;
hadoop: 2.7.0;
cluster in AWS, 10 datanodes: d2.4xlarge
I think it's maybe related with this issue but I'm not using any filters http://apache-hbase.679495.n3.nabble.com/Exception-during-a-reverse-scan-with-filter-td4069721.html
java.lang.RuntimeException: org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of OutOfOrderScannerNextException: was there a rpc timeout?
at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
at com.socialbakers.broker.client.hbase.htable.AbstractHtableListScanner.scanToList(AbstractHtableListScanner.java:30)
at com.socialbakers.broker.client.hbase.htable.AbstractHtableListSingleScanner.invokeOperation(AbstractHtableListSingleScanner.java:23)
at com.socialbakers.broker.client.hbase.htable.AbstractHtableListSingleScanner.invokeOperation(AbstractHtableListSingleScanner.java:11)
at com.socialbakers.broker.client.hbase.AbstractHbaseApi.endPointMethod(AbstractHbaseApi.java:40)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.socialbakers.broker.client.Route.invoke(Route.java:241)
at com.socialbakers.broker.client.handler.EndpointHandler.invoke(EndpointHandler.java:173)
at com.socialbakers.broker.client.handler.EndpointHandler.process(EndpointHandler.java:69)
at com.thetransactioncompany.jsonrpc2.server.Dispatcher.process(Dispatcher.java:196)
at com.socialbakers.broker.client.RejectableRunnable.run(RejectableRunnable.java:38)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of OutOfOrderScannerNextException: was there a rpc timeout?
at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:430)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:333)
at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:91)
... 15 more
Caused by: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 2 But the nextCallSeq got from client: 1; request=scanner_id: 27700695 number_of_rows: 100 close_scanner: false next_call_seq: 1
at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3231)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30946)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at sun.reflect.GeneratedConstructorAccessor16.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:287)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:214)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:115)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:91)
at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:375)
... 17 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException): org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 2 But the nextCallSeq got from client: 1; request=scanner_id: 27700695 number_of_rows: 100 close_scanner: false next_call_seq: 1
at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3231)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30946)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:31392)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:173)
... 21 more