I am trying to read data from an API-endpoint using Ktor and getting an IllegalStateException often regarding supposedly null connection. Here´s how the exception looks like:
2021-04-06 11:14:48.407 [Ktor-client-apache] DEBUG o.a.h.i.n.client.InternalIODispatch - http-outgoing-4 [CLOSED] Exception
java.lang.IllegalStateException: Inconsistent state: managed connection is null
at org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.manageConnectionPersistence(AbstractClientExchangeHandler.java:285)
at org.apache.http.impl.nio.client.MainClientExec.responseCompleted(MainClientExec.java:352)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:172)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
at org.apache.http.impl.nio.client.InternalRequestExecutor.inputReady(InternalRequestExecutor.java:83)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run(Thread.java:748)
Below is my code which causes the exception. I am creating a client on every request to be safe and closing it at the end in addition to checking of it´s active:
suspend fun getData(date: String): MyData? {
val client = HttpClient {
defaultRequest {
contentType(ContentType.Application.Json)
}
install(JsonFeature)
}
var myData: MyData? = null
if (client.isActive) {
try {
val response = client.get<JsonArray> {
url(url)
headers.append("keep-alive", true.toString())
}
val mapper = jacksonObjectMapper()
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
mapper.configure(DeserializationFeature.ACCEPT_EMPTY_ARRAY_AS_NULL_OBJECT, true)
if (response.size() > 0) {
myData =
mapper.readValue(response.get(0).toString(), MyData::class.java)
}
} catch (e: IllegalStateException) {
log.info(e.message)
}
client.close()
}
return myData
}
I need help figuring out the fix for this exception since I wasn´t able to find much documentation about it. Thanks in advance.
Related
I have a Spring Boot project that is leveraging Spring Integration. My goal is to to poll a POP3 mail server regularly, and download any attachments associated with those messages. My relevant Spring Config looks like this:
#Configuration
public class MailIntegrationConfig {
#Value("${path.output.temp}")
private String outPath;
#Bean
public MessageChannel mailChannel() {
return new DirectChannel();
}
#Bean
#InboundChannelAdapter(value = "mailChannel", poller = #Poller(fixedDelay = "16000"))
public MessageSource<Object> fileReadingMessageSource() {
var receiver = new Pop3MailReceiver("pop3s://user:passwordexample.com/INBOX");
var mailProperties = new Properties();
mailProperties.setProperty("mail.pop3.port", "995");
mailProperties.put("mail.pop3.ssl.enable", true);
receiver.setShouldDeleteMessages(false);
receiver.setMaxFetchSize(10);
receiver.setJavaMailProperties(mailProperties);
// receiver.setHeaderMapper(new DefaultMailHeaderMapper());
var source = new MailReceivingMessageSource(receiver);
return source;
}
#Bean
#ServiceActivator(inputChannel = "mailChannel")
public MessageHandler popMessageHandler() {
return new MailReceivingMessageHandler(outPath);
}
}
My MailReceivingMessageHandler class (partial)
public class MailReceivingMessageHandler extends AbstractMessageHandler {
private String outDir;
public MailReceivingMessageHandler(String outDir) {
var outPath = new File(outDir);
if (!outPath.exists()) {
throw new IllegalArgumentException(String.format("%s does not exist.", outDir));
}
this.outDir = outDir;
}
#Override
protected void handleMessageInternal(org.springframework.messaging.Message<?> message) {
Object payload = message.getPayload();
if (!(payload instanceof Message)) {
throw new IllegalArgumentException(
"Unable to create MailMessage from payload type [" + message.getPayload().getClass().getName()
+ "], " + "expected MimeMessage, MailMessage, byte array or String.");
}
try {
var msg = (Message) payload;
System.out.println(String.format("Headers [%s] Subject [%s]. Content-Type [%s].", msg.getAllHeaders(),
msg.getSubject(), msg.getContentType()));
this.handleMessage(msg);
} catch (IOException | MessagingException e) {
e.printStackTrace();
}
}
private void handleMessage(Message msg) throws MessagingException, IOException {
var cType = msg.getContentType();
if (cType.contains(MediaType.TEXT_PLAIN_VALUE)) {
handleText((String) msg.getContent());
} else if (cType.contains(MediaType.MULTIPART_MIXED_VALUE)) {
handleMultipart((Multipart) msg.getContent());
}
}
// See
// https://stackoverflow.com/questions/1748183/download-attachments-using-java-mail
private void handleMultipart(Multipart msgContent) throws MessagingException, IOException {
var mCount = msgContent.getCount();
for (var i = 0; i < mCount; i++) {
this.processAttachments(msgContent.getBodyPart(i));
}
}
private void processAttachments(BodyPart part) throws IOException, MessagingException {
var content = part.getContent();
if (content instanceof InputStream || content instanceof String) {
if (Part.ATTACHMENT.equalsIgnoreCase(part.getDisposition()) || !part.getFileName().isBlank()) {
var fName = String.format("%s.%s", UUID.randomUUID().toString(),
FilenameUtils.getExtension(part.getFileName()));
FileUtils.copyInputStreamToFile(part.getInputStream(), new File(outDir + File.separator + fName));
}
if (content instanceof Multipart) {
Multipart multipart = (Multipart) content;
for (int i = 0; i < multipart.getCount(); i++) {
var bodyPart = multipart.getBodyPart(i);
processAttachments(bodyPart);
}
}
}
}
}
Whenever I run my code using the config above, I receive the following error:
javax.mail.MessagingException: No inputstream from datasource;
nested exception is:
java.lang.IllegalStateException: Folder is not Open
at javax.mail.internet.MimeMultipart.parse(MimeMultipart.java:576)
at javax.mail.internet.MimeMultipart.getCount(MimeMultipart.java:312)
at com.midamcorp.data.mail.MailReceivingMessageHandler.handleMultipart(MailReceivingMessageHandler.java:70)
at com.midamcorp.data.mail.MailReceivingMessageHandler.handleMessage(MailReceivingMessageHandler.java:58)
at com.midamcorp.data.mail.MailReceivingMessageHandler.handleMessageInternal(MailReceivingMessageHandler.java:44)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:62)
at org.springframework.integration.handler.ReplyProducingMessageHandlerWrapper.handleRequestMessage(ReplyProducingMessageHandlerWrapper.java:58)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:134)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:62)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:570)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:520)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.handleMessage(SourcePollingChannelAdapter.java:196)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.messageReceived(AbstractPollingEndpoint.java:444)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:428)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:376)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$null$3(AbstractPollingEndpoint.java:323)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.lambda$execute$0(ErrorHandlingTaskExecutor.java:57)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:55)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$4(AbstractPollingEndpoint.java:320)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Folder is not Open
at com.sun.mail.pop3.POP3Folder.checkOpen(POP3Folder.java:562)
at com.sun.mail.pop3.POP3Folder.getProtocol(POP3Folder.java:592)
at com.sun.mail.pop3.POP3Message.getRawStream(POP3Message.java:154)
at com.sun.mail.pop3.POP3Message.getContentStream(POP3Message.java:251)
at javax.mail.internet.MimePartDataSource.getInputStream(MimePartDataSource.java:78)
at javax.mail.internet.MimeMultipart.parse(MimeMultipart.java:570)
... 35 more
Obviously, the root cause is clear - the POP3 folder is closed. I have seen solutions that would likely be able to handle when just the Java mail classes are used, but none with Spring Integration. My question is how does one properly control when a folder is open or closed using Spring Integration Mail? I realize the Pop3MailReceiver class has a .setAutoCloseFolder() method. Based on the Spring Docs, I assume I need to set that, along something like the following to my handler:
Closeable closeableResource = StaticMessageHeaderAccessor.getCloseableResource(message);
if (closeableResource != null) {
closeableResource.close();
}
However, if I set autoCloseFolder to false, it does not appear as if the message even ever "hits" my handler, so unfortunately being able to close the resource does not even matter at this point. That is, when autoClose is set to false, the 'handleMessageInternal()' method in my handler class is never reached even though there are indeed message on the POP3 server. Instead I just get a bunch of logs like this:
2020-06-26 15:26:54.523 INFO 15348 --- [ scheduling-1] o.s.integration.mail.Pop3MailReceiver : attempting to receive mail from folder [INBOX]
What am I missing?
Thanks.
I have following method:
#Timed(value = "my.request.timer", percentiles = {0.5, 0.95}, histogram = true)
public ResponseEntity<MyResponseDto> executeHttpCall(MyReq myReq) {
log.warn("!!!!! REAL METHOD!!!! for {}", myReq);
Mono<ResponseEntity<MyResponseDto>> responseEntityMono = webClient.post()
.bodyValue(myReq)
.retrieve()
.toEntity(MyResponseDto.class);
try {
return responseEntityMono.block();
} catch (Exception e) {
log.warn("EXCEPTION: ", e);
}
return null;
}
I wanted to add hystrix for that method so I got following:
#HystrixCommand(fallbackMethod = "nullResponse",
threadPoolProperties = {
#HystrixProperty(name = "coreSize", value = "1000"),
#HystrixProperty(name = "maxQueueSize", value = "7777"),
})
#Timed(value = "my.request.timer", percentiles = {0.5, 0.95}, histogram = true)
public ResponseEntity<MyResponseDto> executeHttpCall(MyReq myReq) {
log.warn("!!!!! REAL METHOD!!!! for {}", myReq);
Mono<ResponseEntity<MyResponseDto>> responseEntityMono = webClient.post()
.bodyValue(myReq)
.retrieve()
.toEntity(myResponseDto.class);
try {
return responseEntityMono.block();
} catch (Exception e) {
log.warn("EXCEPTION: ", e);
}
return null;
}
public ResponseEntity<MyResponseDto> nullResponse(MyReq myReq) {
log.warn("Fallback method invoked for {}", myReq);
fallbackMethodIvocationCount.increment();
return null;
}
It became a reason of error(it sometimes reproduces and sometimes - not)
2019-11-11 17:33:51.954 WARN 2996 --- [ConnectorImpl-9] b.m.a.p.MyDetectorAPIConnectorImpl : EXCEPTION:
2019-11-11 17:33:51.954 WARN 2996 --- [ConnectorImpl-9] b.m.a.p.MyDetectorAPIConnectorImpl : EXCEPTION:
reactor.core.Exceptions$ReactiveException: java.lang.InterruptedException
at reactor.core.Exceptions.propagate(Exceptions.java:336)
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:85)
at reactor.core.publisher.Mono.block(Mono.java:1663)
at my.MyService.executeHttpCall(MyDetectorAPIConnectorImpl.java:89)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
What can be reason of this issue and how to fix it ?
It was because of hystrix default timeout. It is possible to disable this timeout to use following property:
hystrix.command.default.execution.timeout.enabled: false
We are building a Graylog output plugin to send data to Google PubSub. This is the code we have written, inspired from the boilerplate code provided by google pubsub (this and this)
try (InputStream credential = new FileInputStream(Objects.requireNonNull(config.getString(CK_CREDENTIAL_FILE)))) {
CredentialsProvider credentialsProvider = FixedCredentialsProvider
.create(ServiceAccountCredentials.fromStream(credential));
// endpoint can be set here
publisher = Publisher.newBuilder(topicName).setCredentialsProvider(credentialsProvider).build();
ByteString finalData = ByteString.copyFromUtf8(String.valueOf(obj));
PubsubMessage pubsubMessage = PubsubMessage.newBuilder()
.setData(finalData)
.build();
ApiFuture<String> future = publisher.publish(pubsubMessage);
messageIdFutures.add(future);
ApiFutures.addCallback(
future,
new ApiFutureCallback<String>() {
#Override
public void onFailure(Throwable throwable) {
if (throwable instanceof ApiException) {
ApiException apiException = ((ApiException) throwable);
// details on the API exception
System.out.println(apiException.getStatusCode().getCode());
System.out.println(apiException.isRetryable());
}
System.out.println("Error publishing message : " + String.valueOf(obj));
}
#Override
public void onSuccess(String messageId) {
// Once published, returns server-assigned message ids (unique within the topic)
System.out.println(messageId);
}
},
MoreExecutors.directExecutor());
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (publisher != null) {
try {
try {
publisher.shutdown();
} catch (Exception e) {
e.printStackTrace();
}
publisher.awaitTermination(1, TimeUnit.MINUTES);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
On running, we get the following error stack:-
java.util.concurrent.ExecutionException: java.lang.IllegalAccessError: tried to access field io.opencensus.trace.unsafe.
ContextUtils.CONTEXT_SPAN_KEY from class io.grpc.internal.CensusTracingModule$TracingClientInterceptor
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:526)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:423)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90)
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:68)
at org.plugin.PubSubOutput.writeBuffer(PubSubOutput.java:159)
at org.plugin.PubSubOutput.write(PubSubOutput.java:85)
at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:191)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:18
1)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
ERROR [AggregateFuture] - Input Future failed with Error - {}
java.lang.IllegalAccessError: tried to access field io.opencensus.trace.unsafe.ContextUtils.CONTEXT_SPAN_KEY from class
io.grpc.internal.CensusTracingModule$TracingClientInterceptor
at io.grpc.internal.CensusTracingModule$TracingClientInterceptor.interceptCall(CensusTracingModule.java:384) ~[g
raylog-plugin-pubsub-output-1.0.0-SNAPSHOT.jar:?]
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156) ~[graylog-plugin-pubsub-ou
tput-1.0.0-SNAPSHOT.jar:?]
at io.grpc.internal.CensusStatsModule$StatsClientInterceptor.interceptCall(CensusStatsModule.java:685) ~[graylog
-plugin-pubsub-output-1.0.0-SNAPSHOT.jar:?]
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156) ~[graylog-plugin-pubsub-ou
tput-1.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.grpc.GrpcHeaderInterceptor.interceptCall(GrpcHeaderInterceptor.java:81) ~[graylog-plugin-p
ubsub-output-1.0.0-SNAPSHOT.jar:?]
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156) ~[graylog-plugin-pubsub-ou
tput-1.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.grpc.GrpcMetadataHandlerInterceptor.interceptCall(GrpcMetadataHandlerInterceptor.java:55)
~[graylog-plugin-pubsub-output-1.0.0-SNAPSHOT.jar:?]
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156) ~[graylog-plugin-pubsub-ou
tput-1.0.0-SNAPSHOT.jar:?]
at io.grpc.internal.ManagedChannelImpl.newCall(ManagedChannelImpl.java:766) ~[graylog-plugin-pubsub-output-1.0.0
-SNAPSHOT.jar:?]
at io.grpc.internal.ForwardingManagedChannel.newCall(ForwardingManagedChannel.java:63) ~[graylog-plugin-pubsub-o
utput-1.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.grpc.ChannelPool.newCall(ChannelPool.java:77) ~[graylog-plugin-pubsub-output-1.0.0-SNAPSHO
T.jar:?]
at com.google.api.gax.grpc.GrpcClientCalls.newCall(GrpcClientCalls.java:88) ~[graylog-plugin-pubsub-output-1.0.0
-SNAPSHOT.jar:?]
at com.google.api.gax.grpc.GrpcDirectCallable.futureCall(GrpcDirectCallable.java:58) ~[graylog-plugin-pubsub-out
put-1.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:64) ~[graylog-plugin-pubs
ub-output-1.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:81) ~[graylog-plugin-pubsub-output-1.0.0-SNA
PSHOT.jar:?]
at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63) ~[graylog-plugin-pubsub-output-1
.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41) ~[graylog-plugin-pubsub-output-1
.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.rpc.BatchingCallable.futureCall(BatchingCallable.java:79) ~[graylog-plugin-pubsub-output-1
.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126) ~[graylog-plugin-pubsub-output-1.0.
0-SNAPSHOT.jar:?]
at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87) ~[graylog-plugin-pubsub-output-1.0.0-S
NAPSHOT.jar:?]
at com.google.cloud.pubsub.v1.Publisher.publishOutstandingBatch(Publisher.java:317) [graylog-plugin-pubsub-outpu
t-1.0.0-SNAPSHOT.jar:?]
at com.google.cloud.pubsub.v1.Publisher.publishAllOutstanding(Publisher.java:306) [graylog-plugin-pubsub-output-
1.0.0-SNAPSHOT.jar:?]
at com.google.cloud.pubsub.v1.Publisher$3.run(Publisher.java:280) [graylog-plugin-pubsub-output-1.0.0-SNAPSHOT.j
ar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.j
ava:180) [?:1.8.0_222]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293
) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: java.lang.IllegalAccessError: tried to access field io.opencensus.trace.unsafe.ContextUtils.CONTEXT_SPAN_KEY
from class io.grpc.internal.CensusTracingModule$TracingClientInterceptor
at io.grpc.internal.CensusTracingModule$TracingClientInterceptor.interceptCall(CensusTracingModule.java:384)
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at io.grpc.internal.CensusStatsModule$StatsClientInterceptor.interceptCall(CensusStatsModule.java:685)
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at com.google.api.gax.grpc.GrpcHeaderInterceptor.interceptCall(GrpcHeaderInterceptor.java:81)
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at com.google.api.gax.grpc.GrpcMetadataHandlerInterceptor.interceptCall(GrpcMetadataHandlerInterceptor.java:55)
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at io.grpc.internal.ManagedChannelImpl.newCall(ManagedChannelImpl.java:766)
at io.grpc.internal.ForwardingManagedChannel.newCall(ForwardingManagedChannel.java:63)
at com.google.api.gax.grpc.ChannelPool.newCall(ChannelPool.java:77)
at com.google.api.gax.grpc.GrpcClientCalls.newCall(GrpcClientCalls.java:88)
at com.google.api.gax.grpc.GrpcDirectCallable.futureCall(GrpcDirectCallable.java:58)
at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:64)
at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:81)
at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63)
at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41)
at com.google.api.gax.rpc.BatchingCallable.futureCall(BatchingCallable.java:79)
at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126)
at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87)
at com.google.cloud.pubsub.v1.Publisher.publishOutstandingBatch(Publisher.java:317)
at com.google.cloud.pubsub.v1.Publisher.publishAllOutstanding(Publisher.java:306)
at com.google.cloud.pubsub.v1.Publisher$3.run(Publisher.java:280)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.j
ava:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293
)
We have even looked at the grpc-java github repo but can't seem to figure out the problem. We are not using grpc in our communication but we think that it is being used by the google pubsub java module internally.
Any help would be highly appreciated. Thanks in advance.
I had the same issue,
My app is developped with scala (with sbt), but it's the same with maven.
To solve the issue I change my dependencies with :
libraryDependencies += "com.google.cloud" % "google-cloud-pubsub" % "1.75.0"
libraryDependencies += "com.google.api-client" % "google-api-client" % "1.30.5"
libraryDependencies += "com.google.cloud" % "google-cloud-core-grpc" % "1.91.3"
For the code, just keep default way to create google credential object :
def publishMessages(projectId : String, topic : String, event : String): Unit = {
val topicName = ProjectTopicName.of(projectId, topic)
var publisher : Publisher = null
val messageIdFutures = new util.ArrayList[ApiFuture[String]]
try {
val credentials = GoogleCredentials.fromStream(new FileInputStream(creds)).createScoped(
java.util.Arrays.asList("https://www.googleapis.com/auth/cloud-platform")
)
publisher = Publisher.newBuilder(topicName)
.setCredentialsProvider(FixedCredentialsProvider.create(credentials)).build
val payload = ByteString.copyFromUtf8(event)
val pubsubMessage = PubsubMessage.newBuilder.setData(payload).build
val messageIdFuture = publisher.publish(pubsubMessage)
messageIdFutures.add(messageIdFuture)
}
I have the following code:
typealias MessagePredicate = (Message) -> Boolean
object EmailHelper {
private val session: Session by lazy {
val props = System.getProperties()
props["mail.imaps.usesocketchannels"] = "true"
props["mail.imap.usesocketchannels"] = "true"
Session.getInstance(props, null)
}
private val store = session.getStore("gimap") as GmailStore
private val idleManager = IdleManager(session, Executors.newSingleThreadExecutor())
private val folder: GmailFolder by lazy { store.getFolder("INBOX") as GmailFolder }
init {
store.connect("imap.gmail.com", "***#gmail.com", "***")
folder.open(Folder.READ_ONLY)
idleManager.watch(folder)
}
fun watchForMessage(condition: MessagePredicate): CompletableFuture<Message> {
val promise = CompletableFuture<Message>()
folder.addMessageCountListener(object : MessageCountAdapter() {
override fun messagesAdded(e: MessageCountEvent) {
super.messagesAdded(e)
e.messages.firstOrNull(condition)?.let {
folder.removeMessageCountListener(this)
promise.complete(it)
}
}
})
return promise
}
}
However when I run this code I'm getting the following exception:
Exception in thread "main" java.lang.ExceptionInInitializerError
at com.muliyul.MainKt.main(Main.kt:28)
Caused by: javax.mail.MessagingException: Folder is not using SocketChannels
at com.sun.mail.imap.IdleManager.watch(IdleManager.java:205)
at com.muliyul.EmailHelper.<clinit>(EmailHelper.kt:40)
... 1 more
I am setting the property "mail.imaps.usesocketchannels" beforehand and I've also read this question yet I can't wrap my head around what's wrong with my code.
Can someone point me in the right direction?
Side note: the email provider is Gmail (obviously).
An hour after I posted this question (and 3 hours of researching) I finally found an answer.
You have to set the property mail.gimap.usesocketchannels to "true" (and not mail.imap.usesocketchannels or mail.imaps.usesocketchannels)
This is due to the fact that gimap is a different protocol than imap.
There goes 3 hours down the drain.
I have completed a "happy-path" (as below).
How I can advise a .transform call to have it invoke an error flow (via errorChannel) w/o interrupting the mainFlow?
Currently the mainFlow terminates on first failure occurrence in second .transform (when payload cannot be deserialized to type). My desired behavior is that I'd like to log and continue processing.
I've read about ExpressionEvaluatingRequestHandlerAdvice. Would I just add a second param to each .transform call like e -> e.advice(myAdviceBean) and declare such a bean with success and error channels? Assuming I'd need to break up my mainFlow to receive success from each transform.
On some commented direction I updated the original code sample. But I'm still having trouble taking this "all the way home".
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.handler.ServiceActivatingHandler DEBUG handler 'ServiceActivator for [org.springframework.integration.dsl.support.BeanNameMessageProcessor#5f3839ad] (org.springframework.integration.handler.ServiceActivatingHandler#0)' produced no reply for request Message: ErrorMessage [payload=org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice$MessageHandlingExpressionEvaluatingAdviceException: Handler Failed; nested exception is org.springframework.integration.transformer.MessageTransformationException: failed to transform message; nested exception is com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "hasDoaCostPriceChanged" (class com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog), not marked as ignorable (18 known properties: "supplierUpdateDate", "fPLOSMaskArrival", "createDate", "endAllowed", "sellStateId", "ratePlanLevel", "ratePlanId", "startAllowed", "stayDate", "doaCostPriceChanged", "hotelId", "logActionTypeId" [truncated]])
at [Source: java.util.zip.GZIPInputStream#242017b8; line: 1, column: 32] (through reference chain: com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog["hasDoaCostPriceChanged"]), headers={id=c054d976-5750-827f-8894-51aba9655c77, timestamp=1441738159660}]
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.channel.DirectChannel DEBUG postSend (sent=true) on channel 'errorChannel', message: ErrorMessage [payload=org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice$MessageHandlingExpressionEvaluatingAdviceException: Handler Failed; nested exception is org.springframework.integration.transformer.MessageTransformationException: failed to transform message; nested exception is com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "hasDoaCostPriceChanged" (class com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog), not marked as ignorable (18 known properties: "supplierUpdateDate", "fPLOSMaskArrival", "createDate", "endAllowed", "sellStateId", "ratePlanLevel", "ratePlanId", "startAllowed", "stayDate", "doaCostPriceChanged", "hotelId", "logActionTypeId" [truncated]])
at [Source: java.util.zip.GZIPInputStream#242017b8; line: 1, column: 32] (through reference chain: com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog["hasDoaCostPriceChanged"]), headers={id=c054d976-5750-827f-8894-51aba9655c77, timestamp=1441738159660}]
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.channel.DirectChannel DEBUG preSend on channel 'mainFlow.channel#3', message: GenericMessage [payload=java.util.zip.GZIPInputStream#242017b8, headers={id=b80106f9-7f4c-1b92-6aca-6e73d3bf8792, timestamp=1441738159664}]
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.aggregator.AggregatingMessageHandler DEBUG org.springframework.integration.aggregator.AggregatingMessageHandler#0 received message: GenericMessage [payload=java.util.zip.GZIPInputStream#242017b8, headers={id=b80106f9-7f4c-1b92-6aca-6e73d3bf8792, timestamp=1441738159664}]
2015-09-08 11:49:19,665 [pool-3-thread-1] org.springframework.integration.channel.DirectChannel DEBUG preSend on channel 'errorChannel', message: ErrorMessage [payload=org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.aggregator.AggregatingMessageHandler#0]; nested exception is java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?, headers={id=24e3a1c7-af6b-032c-6a29-b55031fba0d7, timestamp=1441738159665}]
2015-09-08 11:49:19,665 [pool-3-thread-1] org.springframework.integration.handler.ServiceActivatingHandler DEBUG ServiceActivator for [org.springframework.integration.dsl.support.BeanNameMessageProcessor#5f3839ad] (org.springframework.integration.handler.ServiceActivatingHandler#0) received message: ErrorMessage [payload=org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.aggregator.AggregatingMessageHandler#0]; nested exception is java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?, headers={id=24e3a1c7-af6b-032c-6a29-b55031fba0d7, timestamp=1441738159665}]
2015-09-08 11:49:19,665 [pool-3-thread-1] com.xxx.DataMigrationModule$ErrorService ERROR org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.aggregator.AggregatingMessageHandler#0]; nested exception is java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:84)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:287)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:245)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:95)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:231)
at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:154)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:102)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:105)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.access$000(UnicastingDispatcher.java:48)
at org.springframework.integration.dispatcher.UnicastingDispatcher$1.run(UnicastingDispatcher.java:92)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?
at org.springframework.util.Assert.state(Assert.java:385)
at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.handleMessageInternal(AbstractCorrelatingMessageHandler.java:369)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
... 22 more
UPDATED (09-08-2015)
code sample
#Bean
public IntegrationFlow mainFlow() {
// #formatter:off
return IntegrationFlows
.from(
amazonS3InboundSynchronizationMessageSource(),
e -> e.poller(p -> p.trigger(this::nextExecutionTime))
)
.transform(unzipTransformer())
.split(f -> new FileSplitter())
.channel(MessageChannels.executor(Executors.newCachedThreadPool()))
.transform(Transformers.fromJson(persistentType()), , e -> e.advice(handlingAdvice()))
// #see http://docs.spring.io/spring-integration/reference/html/messaging-routing-chapter.html#agg-and-group-to
.aggregate(a ->
a.releaseStrategy(g -> g.size() == persistenceBatchSize)
.expireGroupsUponCompletion(true)
.sendPartialResultOnExpiry(true)
.groupTimeoutExpression("size() ge 2 ? 10000 : -1")
, null
)
.handle(jdbcRepositoryHandler())
// TODO add advised PollableChannel to deal with possible persistence issue and retry with partial batch
.get();
// #formatter:on
}
#Bean
public ErrorService errorService() {
return new ErrorService();
}
#Bean
public MessageChannel customErrorChannel() {
return MessageChannels.direct().get();
}
#Bean
public IntegrationFlow customErrorFlow() {
// #formatter:off
return IntegrationFlows
.from(customErrorChannel())
.handle("errorService", "handleError")
.get();
// #formatter:on
}
#Bean
ExpressionEvaluatingRequestHandlerAdvice handlingAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setOnFailureExpression("payload");
advice.setFailureChannel(customErrorChannel());
advice.setReturnFailureExpressionResult(true);
advice.setTrapException(true);
return advice;
}
protected class ErrorService implements ErrorHandler {
private final Logger log = LoggerFactory.getLogger(getClass());
#Override
public void handleError(Throwable t) {
stopEndpoints(t);
}
private void stopEndpoints(Throwable t) {
log.error(ExceptionUtils.getStackTrace(t));
}
}
Turns out I had things wrong in a few places, like:
I had to autowire a Jackson2 ObjectMapper (that I get from Sprint Boot auto-config) and construct an instance of JsonObjectMapper to be added as second arg in Transformers.fromJson; made for more lenient unmarshalling to persistent type (stops UnrecognizedPropertyException); and thus waived need for ExpressionEvaluatingRequestHandlerAdvice
Choosing the proper variant of .split method in IntegrationFlowDefinition in order to employ the FileSplitter, otherwise you don't get this splitter rather a DefaultMessageSplitter which pre-maturely terminates flow after first record read from InputStream
Moved transform, aggregate, handle to a its own pubsub channel employing an async task executor
Still not 100% of what I need, but it's much further along.
See what I ended up w/ below...
#Configuration
#EnableIntegration
#IntegrationComponentScan
public class DataMigrationModule {
private final Logger log = LoggerFactory.getLogger(getClass());
#Value("${cloud.aws.credentials.accessKey}")
private String accessKey;
#Value("${cloud.aws.credentials.secretKey}")
private String secretKey;
#Value("${cloud.aws.s3.bucket}")
private String bucket;
#Value("${cloud.aws.s3.max-objects-per-batch:1024}")
private int maxObjectsPerBatch;
#Value("${cloud.aws.s3.accept-subfolders:false}")
private String acceptSubFolders;
#Value("${cloud.aws.s3.remote-directory}")
private String remoteDirectory;
#Value("${cloud.aws.s3.local-directory-ref:java.io.tmpdir}")
private String localDirectoryRef;
#Value("${cloud.aws.s3.local-subdirectory:target/s3-dump}")
private String localSubdirectory;
#Value("${cloud.aws.s3.filename-wildcard:}")
private String fileNameWildcard;
#Value("${app.persistent-type:}")
private String persistentType;
#Value("${app.repository-type:}")
private String repositoryType;
#Value("${app.persistence-batch-size:2500}")
private int persistenceBatchSize;
#Value("${app.persistence-batch-release-timeout-in-milliseconds:5000}")
private int persistenceBatchReleaseTimeoutMillis;
#Autowired
private ListableBeanFactory beanFactory;
#Autowired
private ObjectMapper objectMapper;
private final AtomicBoolean invoked = new AtomicBoolean();
private Class<?> repositoryType() {
try {
return Class.forName(repositoryType);
} catch (ClassNotFoundException cnfe) {
log.error("Unknown repository implementation!", cnfe);
System.exit(0);
}
return null;
}
private Class<?> persistentType() {
try {
return Class.forName(persistentType);
} catch (ClassNotFoundException cnfe) {
log.error("Unsupported type!", cnfe);
System.exit(0);
}
return null;
}
public Date nextExecutionTime(TriggerContext triggerContext) {
return this.invoked.getAndSet(true) ? null : new Date();
}
#Bean
public FileToInputStreamTransformer unzipTransformer() {
FileToInputStreamTransformer transformer = new FileToInputStreamTransformer();
transformer.setDeleteFiles(true);
return transformer;
}
#Bean
public MessageSource<?> amazonS3InboundSynchronizationMessageSource() {
AWSCredentials credentials = new BasicAWSCredentials(this.accessKey, this.secretKey);
AmazonS3InboundSynchronizationMessageSource messageSource = new AmazonS3InboundSynchronizationMessageSource();
messageSource.setCredentials(credentials);
messageSource.setBucket(bucket);
messageSource.setMaxObjectsPerBatch(maxObjectsPerBatch);
messageSource.setAcceptSubFolders(Boolean.valueOf(acceptSubFolders));
messageSource.setRemoteDirectory(remoteDirectory);
if (!fileNameWildcard.isEmpty()) {
messageSource.setFileNameWildcard(fileNameWildcard);
}
String directory = System.getProperty(localDirectoryRef);
if (!localSubdirectory.startsWith("/")) {
localSubdirectory = "/" + localSubdirectory;
}
if (!localSubdirectory.endsWith("/")) {
localSubdirectory = localSubdirectory + "/";
}
directory = directory + localSubdirectory;
FileUtils.mkdir(directory);
messageSource.setDirectory(new LiteralExpression(directory));
return messageSource;
}
#Bean
public IntegrationFlow mainFlow() {
// #formatter:off
return IntegrationFlows
.from(
amazonS3InboundSynchronizationMessageSource(),
e -> e.poller(p -> p.trigger(this::nextExecutionTime))
)
.transform(unzipTransformer())
.split(new FileSplitter(), null)
.publishSubscribeChannel(new SimpleAsyncTaskExecutor(), p -> p.subscribe(persistenceSubFlow()))
.get();
// #formatter:on
}
#Bean
public IntegrationFlow persistenceSubFlow() {
JsonObjectMapper<?, ?> jsonObjectMapper = new Jackson2JsonObjectMapper(objectMapper);
ReleaseStrategy releaseStrategy = new TimeoutCountSequenceSizeReleaseStrategy(persistenceBatchSize,
persistenceBatchReleaseTimeoutMillis);
// #formatter:off
return f -> f
.transform(Transformers.fromJson(persistentType(), jsonObjectMapper))
// #see http://docs.spring.io/spring-integration/reference/html/messaging-routing-chapter.html#agg-and-group-to
.aggregate(
a -> a
.releaseStrategy(releaseStrategy)
.correlationStrategy(m -> m.getHeaders().get("id"))
.expireGroupsUponCompletion(true)
.sendPartialResultOnExpiry(true)
, null
)
.handle(jdbcRepositoryHandler());
// #formatter:on
}
#Bean
public JdbcRepositoryHandler jdbcRepositoryHandler() {
return new JdbcRepositoryHandler(repositoryType(), beanFactory);
}
protected class JdbcRepositoryHandler extends AbstractMessageHandler {
#SuppressWarnings("rawtypes")
private Insertable repository;
public JdbcRepositoryHandler(Class<?> repositoryClass, ListableBeanFactory beanFactory) {
repository = (Insertable<?>) beanFactory.getBean(repositoryClass);
}
#Override
protected void handleMessageInternal(Message<?> message) {
repository.insert((List<?>) message.getPayload());
}
}
protected class FileToInputStreamTransformer extends AbstractFilePayloadTransformer<InputStream> {
#Override
protected InputStream transformFile(File payload) throws Exception {
return new GZIPInputStream(new FileInputStream(payload));
}
}
}
Yes, you are correct. To advice the handle() method of Transformer's MessageHandler you should use exactly that e.advice method of the second parameter of .transform() EIP-method. And yes: you should define ExpressionEvaluatingRequestHandlerAdvice bean for your purpose.
You can reuse that Advice bean for different goals to handle successes and failures the same manner.
UPDATE
Although it isn't clear to me how you'd like to continue the flow with the wrong message, but you you can use onFailureExpression and returnFailureExpressionResult=true of the ExpressionEvaluatingRequestHandlerAdvice to return something after the unzipErrorChannel().
BTW the failureChannel logic doesn't work without onFailureExpression:
if (this.onFailureExpression != null) {
Object evalResult = this.evaluateFailureExpression(message, actualException);
if (this.returnFailureExpressionResult) {
return evalResult;
}
}