I am running a java application on kubernetes which upload multiple files from local container to s3 bucket using s3a, but I am getting the below exception in logs and files are not getting uploaded to s3.
Partial files are getting uploaded to s3
2021-11-30 12:28:44,529 614982 [secor-threadpool-1] ERROR c.pinterest.secor.uploader.Uploader - Error while uploading to S3 : java.lang.RuntimeException: org.apache.hadoop.fs.s3a.AWSClientIOException: copyFromLocalFile(/tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate, s3a://prod-dataplatform-hive/orc_secor_test/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate) on /tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate: com.amazonaws.AmazonClientException: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V
java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.hadoop.fs.s3a.AWSClientIOException: copyFromLocalFile(/tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate, s3a://prod-dataplatform-hive/orc_secor_test/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate) on /tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate: com.amazonaws.AmazonClientException: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.pinterest.secor.uploader.FutureHandle.get(FutureHandle.java:34)
at com.pinterest.secor.uploader.Uploader.uploadFiles(Uploader.java:117)
at com.pinterest.secor.uploader.Uploader.checkTopicPartition(Uploader.java:244)
at com.pinterest.secor.uploader.Uploader.applyPolicy(Uploader.java:284)
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:141)
at com.pinterest.secor.consumer.Consumer.run(Consumer.java:133)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.hadoop.fs.s3a.AWSClientIOException: copyFromLocalFile(/tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate, s3a://prod-dataplatform-hive/orc_secor_test/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate) on /tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate: com.amazonaws.AmazonClientException: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V
at com.pinterest.secor.uploader.HadoopS3UploadManager$1.run(HadoopS3UploadManager.java:63)
... 5 more
Caused by: org.apache.hadoop.fs.s3a.AWSClientIOException: copyFromLocalFile(/tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate, s3a://prod-dataplatform-hive/orc_secor_test/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate) on /tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate: com.amazonaws.AmazonClientException: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:144)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:117)
at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFromLocalFile(S3AFileSystem.java:2039)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2320)
at org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:2307)
at com.pinterest.secor.util.FileUtil.moveToCloud(FileUtil.java:206)
at com.pinterest.secor.uploader.HadoopS3UploadManager$1.run(HadoopS3UploadManager.java:61)
... 5 more
Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V
at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:286)
at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:265)
at com.amazonaws.services.s3.transfer.internal.UploadImpl.waitForUploadResult(UploadImpl.java:66)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerCopyFromLocalFile(S3AFileSystem.java:2099)
at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFromLocalFile(S3AFileSystem.java:2037)
... 9 more
Caused by: java.lang.NoSuchMethodError: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V
at com.amazonaws.services.s3.model.S3DataSource$Utils.cleanupDataSource(S3DataSource.java:48)
at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1863)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1817)
at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:169)
at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:149)
at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:115)
at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:45)
... 4 more
Looks like this is a bug.
problems caused by AWS SDK classes from the localstack-utils-fat.jar overriding classes defined in the actual Lambda jar/zip itself.
Here's the version you need with the fix. It sounds like there is a work around:
implemented a partial fix for this that moved the localstack-utils-fat.jar later on the classpath, but this fix only applied to lambdas being run using the docker executor.
Basically, it's not your fault. It's a code issue with dependencies overwriting each others function signatures. You need to use the latest localstack-utils-fat.jar and it should fix your issue.
Related
I would like to use Spring Cloud Stream kinesis binder with KPL/KCL enabled. However, when I enabled that by using kpl-kcl-enabled: true the following error keeps coming up:
com.amazonaws.services.kinesis.producer.IrrecoverableError: Error starting child process at
com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:537) at
com.amazonaws.services.kinesis.producer.Daemon.startChildProcess(Daemon.java:468) at
com.amazonaws.services.kinesis.producer.Daemon.access$100(Daemon.java:63) at
com.amazonaws.services.kinesis.producer.Daemon$1.run(Daemon.java:133) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at
java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Cannot run program "/tmp/amazon-kinesis-producer-native-
binaries/kinesis_producer_685427917724EC847D7D65F261E7040F3FCCB039": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) at
com.amazonaws.services.kinesis.producer.Daemon.startChildProcess(Daemon.java:466) ... 5 common frames omitted Caused by: java.io.IOException: error=2, No such file or directory at
java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:247) at java.lang.ProcessImpl.start(ProcessImpl.java:134) at
java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ... 6 common frames omitted
After quite a few attempts to restart it it throws out of memory exception:
Exception in thread "kpl-daemon-0000" java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at com.amazonaws.services.kinesis.producer.Daemon.<init>(Daemon.java:95)
at com.amazonaws.services.kinesis.producer.KinesisProducer$MessageHandler.onError(KinesisProducer.java:168)
at com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:537)
at com.amazonaws.services.kinesis.producer.Daemon.startChildProcess(Daemon.java:468)
at com.amazonaws.services.kinesis.producer.Daemon.access$100(Daemon.java:63)
at com.amazonaws.services.kinesis.producer.Daemon$1.run(Daemon.java:133)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
KPL expects glibc in version 2.5 or higher to be available in your Linux version.
openjdk:8-jdk-alpine docker image does not provide that.
You need to use a different docker image, for example: openjdk:8-jdk-slim to have jdk and glibc already installed, or frolvlad/alpine-glibc for alpine image with glibc.
I'm trying to run Neo4j database with java but it gives me the following error :
java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, /home/matteo/neo4j-community-2.2.2/data/graph.db
at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:334)
at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:59)
at org.neo4j.graphdb.factory.GraphDatabaseFactory.newDatabase(GraphDatabaseFactory.java:108)
at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:95)
at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:176)
at org.neo4j.graphdb.factory.GraphDatabaseFactory.newEmbeddedDatabase(GraphDatabaseFactory.java:67)
at Getter.main(Getter.java:46)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.extension.KernelExtensions#2e9a2123' failed to initialize. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:489)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:72)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:106)
at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:330)
... 6 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.index.lucene.LuceneKernelExtension#1c3c1d18' failed to initialize. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:489)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:72)
at org.neo4j.kernel.extension.KernelExtensions.init(KernelExtensions.java:74)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:483)
... 9 more
Caused by: java.lang.NoSuchFieldError: LUCENE_36
at org.neo4j.index.impl.lucene.LuceneDataSource.<clinit>(LuceneDataSource.java:88)
at org.neo4j.index.lucene.LuceneKernelExtension.init(LuceneKernelExtension.java:52)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:483)
... 12 more
Where is the problem??
Thank you very much!
Matteo
Since you're running embedded Neo4j this looks like you're missing lucene-core-3.6.2.jar on your classpath.
Best practice is to use a build system supporting transitive dependencies like gradle or maven. For those the most easy this is have a dependency on neo4j-community artifact, e.g. http://mvnrepository.com/artifact/org.neo4j/neo4j/2.2.5. This will pull in all other required jars.
I'm having issues on getting Solr 4.3.1 to run using the cargo-maven2-plugin. After fixing all classpath issues, I've ended up with the following error:
ERROR org.apache.solr.core.CoreContainer - null:org.apache.solr.common.SolrException: Unable to create core: collection1
at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1450)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:993)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597)
<...>
Caused by: org.apache.solr.common.SolrException: RequestHandler init failure
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:821)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:618)
at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984)
... 10 more
Caused by: org.apache.solr.common.SolrException: RequestHandler init failure
at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:168)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:758)
... 13 more
Caused by: org.apache.solr.common.SolrException: Error Instantiating Request Handler, org.apache.solr.handler.dataimport.DataImportHandler failed to instantiate org.apache.solr.request.SolrRequestHandler
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:539)
at org.apache.solr.core.SolrCore.createRequestHandler(SolrCore.java:592)
at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:154)
... 14 more
Caused by: java.lang.ClassCastException: class org.apache.solr.handler.dataimport.DataImportHandler
at java.lang.Class.asSubclass(Class.java:3037)
at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:448)
at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:396)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:518)
... 16 more
I've tried with the tomcat7x and jetty7x containers. If this is a classloader problem, can I somehow force the container to use the same classloader as maven is using?
Going through the stack trace I did a Google search on
Error Instantiating Request Handler, \
org.apache.solr.handler.dataimport.DataImportHandler \
failed to instantiate org.apache.solr.request.SolrRequestHandl
And that brought up the following SO question.
Why can't I instantiate DataImportHandler in Solr on JBoss?
There are many answers that should help resolve the issue.
MemcacheD was working fine until we recently switched over to Java 7 from Java 6. Now we keep getting exceptions such as the following:
java.lang.RuntimeException: Exception waiting for value
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1025)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1042)
...
...
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Cancelled
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:169)
at net.spy.memcached.internal.GetFuture.get(GetFuture.java:62)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1021)
and this
java.lang.RuntimeException: Failed waiting for store
at net.spy.memcached.MemcachedClient.mutateWithDefault(MemcachedClient.java:1696)
at net.spy.memcached.MemcachedClient.incr(MemcachedClient.java:1806)
...
...
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Cancelled
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:169)
at net.spy.memcached.MemcachedClient.mutateWithDefault(MemcachedClient.java:1687)
... 75 more
thank you are helping!
I was trying to start Cassandra on FreeBSD with /usr/local/share/cassandra/bin/cassandra.
But it gives me the following error messages, any idea how to get around the problem? Any suggestions will be GREATLY appreciated. Thank you.
INFO 21:16:38,666 JNA not found. Native methods will be disabled.
ERROR 21:16:38,684 Exception encountered during startup.
java.lang.ExceptionInInitializerError
at org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:73)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:224)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Cannot locate storage-conf.xml via storage-config system property or classpath lookup.
at org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:584)
... 2 more
Caused by: java.lang.RuntimeException: Cannot locate storage-conf.xml via storage-config system property or classpath lookup.
at org.apache.cassandra.config.DatabaseDescriptor.getStorageConfigPath(DatabaseDescriptor.java:180)
at org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:187)
... 2 more
Exception encountered during startup.
java.lang.ExceptionInInitializerError
at org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:73)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:224)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Cannot locate storage-conf.xml via storage-config system property or classpath lookup.
at org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:584)
... 2 more
Caused by: java.lang.RuntimeException: Cannot locate storage-conf.xml via storage-config system property or classpath lookup.
at org.apache.cassandra.config.DatabaseDescriptor.getStorageConfigPath(DatabaseDescriptor.java:180)
at org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:187)
... 2 more
Is this cassandra 0.6.x? You probably want to upgrade to the latest cassandra, 0.7.4
The error is because Cassandra can't locate storage-conf.xml. This file should be in your cassandra conf directory, see http://wiki.apache.org/cassandra/StorageConfiguration
If you upgrade to cassandra 0.7.x, you will need a cassandra.yaml file instead.