Graylog Google Pub/Sub output plugin issue - java

We are building a Graylog output plugin to send data to Google PubSub. This is the code we have written, inspired from the boilerplate code provided by google pubsub (this and this)
try (InputStream credential = new FileInputStream(Objects.requireNonNull(config.getString(CK_CREDENTIAL_FILE)))) {
CredentialsProvider credentialsProvider = FixedCredentialsProvider
.create(ServiceAccountCredentials.fromStream(credential));
// endpoint can be set here
publisher = Publisher.newBuilder(topicName).setCredentialsProvider(credentialsProvider).build();
ByteString finalData = ByteString.copyFromUtf8(String.valueOf(obj));
PubsubMessage pubsubMessage = PubsubMessage.newBuilder()
.setData(finalData)
.build();
ApiFuture<String> future = publisher.publish(pubsubMessage);
messageIdFutures.add(future);
ApiFutures.addCallback(
future,
new ApiFutureCallback<String>() {
#Override
public void onFailure(Throwable throwable) {
if (throwable instanceof ApiException) {
ApiException apiException = ((ApiException) throwable);
// details on the API exception
System.out.println(apiException.getStatusCode().getCode());
System.out.println(apiException.isRetryable());
}
System.out.println("Error publishing message : " + String.valueOf(obj));
}
#Override
public void onSuccess(String messageId) {
// Once published, returns server-assigned message ids (unique within the topic)
System.out.println(messageId);
}
},
MoreExecutors.directExecutor());
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (publisher != null) {
try {
try {
publisher.shutdown();
} catch (Exception e) {
e.printStackTrace();
}
publisher.awaitTermination(1, TimeUnit.MINUTES);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
On running, we get the following error stack:-
java.util.concurrent.ExecutionException: java.lang.IllegalAccessError: tried to access field io.opencensus.trace.unsafe.
ContextUtils.CONTEXT_SPAN_KEY from class io.grpc.internal.CensusTracingModule$TracingClientInterceptor
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:526)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:423)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90)
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:68)
at org.plugin.PubSubOutput.writeBuffer(PubSubOutput.java:159)
at org.plugin.PubSubOutput.write(PubSubOutput.java:85)
at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:191)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:18
1)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
ERROR [AggregateFuture] - Input Future failed with Error - {}
java.lang.IllegalAccessError: tried to access field io.opencensus.trace.unsafe.ContextUtils.CONTEXT_SPAN_KEY from class
io.grpc.internal.CensusTracingModule$TracingClientInterceptor
at io.grpc.internal.CensusTracingModule$TracingClientInterceptor.interceptCall(CensusTracingModule.java:384) ~[g
raylog-plugin-pubsub-output-1.0.0-SNAPSHOT.jar:?]
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156) ~[graylog-plugin-pubsub-ou
tput-1.0.0-SNAPSHOT.jar:?]
at io.grpc.internal.CensusStatsModule$StatsClientInterceptor.interceptCall(CensusStatsModule.java:685) ~[graylog
-plugin-pubsub-output-1.0.0-SNAPSHOT.jar:?]
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156) ~[graylog-plugin-pubsub-ou
tput-1.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.grpc.GrpcHeaderInterceptor.interceptCall(GrpcHeaderInterceptor.java:81) ~[graylog-plugin-p
ubsub-output-1.0.0-SNAPSHOT.jar:?]
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156) ~[graylog-plugin-pubsub-ou
tput-1.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.grpc.GrpcMetadataHandlerInterceptor.interceptCall(GrpcMetadataHandlerInterceptor.java:55)
~[graylog-plugin-pubsub-output-1.0.0-SNAPSHOT.jar:?]
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156) ~[graylog-plugin-pubsub-ou
tput-1.0.0-SNAPSHOT.jar:?]
at io.grpc.internal.ManagedChannelImpl.newCall(ManagedChannelImpl.java:766) ~[graylog-plugin-pubsub-output-1.0.0
-SNAPSHOT.jar:?]
at io.grpc.internal.ForwardingManagedChannel.newCall(ForwardingManagedChannel.java:63) ~[graylog-plugin-pubsub-o
utput-1.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.grpc.ChannelPool.newCall(ChannelPool.java:77) ~[graylog-plugin-pubsub-output-1.0.0-SNAPSHO
T.jar:?]
at com.google.api.gax.grpc.GrpcClientCalls.newCall(GrpcClientCalls.java:88) ~[graylog-plugin-pubsub-output-1.0.0
-SNAPSHOT.jar:?]
at com.google.api.gax.grpc.GrpcDirectCallable.futureCall(GrpcDirectCallable.java:58) ~[graylog-plugin-pubsub-out
put-1.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:64) ~[graylog-plugin-pubs
ub-output-1.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:81) ~[graylog-plugin-pubsub-output-1.0.0-SNA
PSHOT.jar:?]
at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63) ~[graylog-plugin-pubsub-output-1
.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41) ~[graylog-plugin-pubsub-output-1
.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.rpc.BatchingCallable.futureCall(BatchingCallable.java:79) ~[graylog-plugin-pubsub-output-1
.0.0-SNAPSHOT.jar:?]
at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126) ~[graylog-plugin-pubsub-output-1.0.
0-SNAPSHOT.jar:?]
at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87) ~[graylog-plugin-pubsub-output-1.0.0-S
NAPSHOT.jar:?]
at com.google.cloud.pubsub.v1.Publisher.publishOutstandingBatch(Publisher.java:317) [graylog-plugin-pubsub-outpu
t-1.0.0-SNAPSHOT.jar:?]
at com.google.cloud.pubsub.v1.Publisher.publishAllOutstanding(Publisher.java:306) [graylog-plugin-pubsub-output-
1.0.0-SNAPSHOT.jar:?]
at com.google.cloud.pubsub.v1.Publisher$3.run(Publisher.java:280) [graylog-plugin-pubsub-output-1.0.0-SNAPSHOT.j
ar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.j
ava:180) [?:1.8.0_222]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293
) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: java.lang.IllegalAccessError: tried to access field io.opencensus.trace.unsafe.ContextUtils.CONTEXT_SPAN_KEY
from class io.grpc.internal.CensusTracingModule$TracingClientInterceptor
at io.grpc.internal.CensusTracingModule$TracingClientInterceptor.interceptCall(CensusTracingModule.java:384)
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at io.grpc.internal.CensusStatsModule$StatsClientInterceptor.interceptCall(CensusStatsModule.java:685)
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at com.google.api.gax.grpc.GrpcHeaderInterceptor.interceptCall(GrpcHeaderInterceptor.java:81)
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at com.google.api.gax.grpc.GrpcMetadataHandlerInterceptor.interceptCall(GrpcMetadataHandlerInterceptor.java:55)
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at io.grpc.internal.ManagedChannelImpl.newCall(ManagedChannelImpl.java:766)
at io.grpc.internal.ForwardingManagedChannel.newCall(ForwardingManagedChannel.java:63)
at com.google.api.gax.grpc.ChannelPool.newCall(ChannelPool.java:77)
at com.google.api.gax.grpc.GrpcClientCalls.newCall(GrpcClientCalls.java:88)
at com.google.api.gax.grpc.GrpcDirectCallable.futureCall(GrpcDirectCallable.java:58)
at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:64)
at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:81)
at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63)
at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41)
at com.google.api.gax.rpc.BatchingCallable.futureCall(BatchingCallable.java:79)
at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126)
at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87)
at com.google.cloud.pubsub.v1.Publisher.publishOutstandingBatch(Publisher.java:317)
at com.google.cloud.pubsub.v1.Publisher.publishAllOutstanding(Publisher.java:306)
at com.google.cloud.pubsub.v1.Publisher$3.run(Publisher.java:280)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.j
ava:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293
)
We have even looked at the grpc-java github repo but can't seem to figure out the problem. We are not using grpc in our communication but we think that it is being used by the google pubsub java module internally.
Any help would be highly appreciated. Thanks in advance.

I had the same issue,
My app is developped with scala (with sbt), but it's the same with maven.
To solve the issue I change my dependencies with :
libraryDependencies += "com.google.cloud" % "google-cloud-pubsub" % "1.75.0"
libraryDependencies += "com.google.api-client" % "google-api-client" % "1.30.5"
libraryDependencies += "com.google.cloud" % "google-cloud-core-grpc" % "1.91.3"
For the code, just keep default way to create google credential object :
def publishMessages(projectId : String, topic : String, event : String): Unit = {
val topicName = ProjectTopicName.of(projectId, topic)
var publisher : Publisher = null
val messageIdFutures = new util.ArrayList[ApiFuture[String]]
try {
val credentials = GoogleCredentials.fromStream(new FileInputStream(creds)).createScoped(
java.util.Arrays.asList("https://www.googleapis.com/auth/cloud-platform")
)
publisher = Publisher.newBuilder(topicName)
.setCredentialsProvider(FixedCredentialsProvider.create(credentials)).build
val payload = ByteString.copyFromUtf8(event)
val pubsubMessage = PubsubMessage.newBuilder.setData(payload).build
val messageIdFuture = publisher.publish(pubsubMessage)
messageIdFutures.add(messageIdFuture)
}

Related

Null connection in Ktor causing java.lang.IllegalStateException:

I am trying to read data from an API-endpoint using Ktor and getting an IllegalStateException often regarding supposedly null connection. Here´s how the exception looks like:
2021-04-06 11:14:48.407 [Ktor-client-apache] DEBUG o.a.h.i.n.client.InternalIODispatch - http-outgoing-4 [CLOSED] Exception
java.lang.IllegalStateException: Inconsistent state: managed connection is null
at org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.manageConnectionPersistence(AbstractClientExchangeHandler.java:285)
at org.apache.http.impl.nio.client.MainClientExec.responseCompleted(MainClientExec.java:352)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:172)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
at org.apache.http.impl.nio.client.InternalRequestExecutor.inputReady(InternalRequestExecutor.java:83)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run(Thread.java:748)
Below is my code which causes the exception. I am creating a client on every request to be safe and closing it at the end in addition to checking of it´s active:
suspend fun getData(date: String): MyData? {
val client = HttpClient {
defaultRequest {
contentType(ContentType.Application.Json)
}
install(JsonFeature)
}
var myData: MyData? = null
if (client.isActive) {
try {
val response = client.get<JsonArray> {
url(url)
headers.append("keep-alive", true.toString())
}
val mapper = jacksonObjectMapper()
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
mapper.configure(DeserializationFeature.ACCEPT_EMPTY_ARRAY_AS_NULL_OBJECT, true)
if (response.size() > 0) {
myData =
mapper.readValue(response.get(0).toString(), MyData::class.java)
}
} catch (e: IllegalStateException) {
log.info(e.message)
}
client.close()
}
return myData
}
I need help figuring out the fix for this exception since I wasn´t able to find much documentation about it. Thanks in advance.

Download Attachments with Spring Integration and POP3

I have a Spring Boot project that is leveraging Spring Integration. My goal is to to poll a POP3 mail server regularly, and download any attachments associated with those messages. My relevant Spring Config looks like this:
#Configuration
public class MailIntegrationConfig {
#Value("${path.output.temp}")
private String outPath;
#Bean
public MessageChannel mailChannel() {
return new DirectChannel();
}
#Bean
#InboundChannelAdapter(value = "mailChannel", poller = #Poller(fixedDelay = "16000"))
public MessageSource<Object> fileReadingMessageSource() {
var receiver = new Pop3MailReceiver("pop3s://user:passwordexample.com/INBOX");
var mailProperties = new Properties();
mailProperties.setProperty("mail.pop3.port", "995");
mailProperties.put("mail.pop3.ssl.enable", true);
receiver.setShouldDeleteMessages(false);
receiver.setMaxFetchSize(10);
receiver.setJavaMailProperties(mailProperties);
// receiver.setHeaderMapper(new DefaultMailHeaderMapper());
var source = new MailReceivingMessageSource(receiver);
return source;
}
#Bean
#ServiceActivator(inputChannel = "mailChannel")
public MessageHandler popMessageHandler() {
return new MailReceivingMessageHandler(outPath);
}
}
My MailReceivingMessageHandler class (partial)
public class MailReceivingMessageHandler extends AbstractMessageHandler {
private String outDir;
public MailReceivingMessageHandler(String outDir) {
var outPath = new File(outDir);
if (!outPath.exists()) {
throw new IllegalArgumentException(String.format("%s does not exist.", outDir));
}
this.outDir = outDir;
}
#Override
protected void handleMessageInternal(org.springframework.messaging.Message<?> message) {
Object payload = message.getPayload();
if (!(payload instanceof Message)) {
throw new IllegalArgumentException(
"Unable to create MailMessage from payload type [" + message.getPayload().getClass().getName()
+ "], " + "expected MimeMessage, MailMessage, byte array or String.");
}
try {
var msg = (Message) payload;
System.out.println(String.format("Headers [%s] Subject [%s]. Content-Type [%s].", msg.getAllHeaders(),
msg.getSubject(), msg.getContentType()));
this.handleMessage(msg);
} catch (IOException | MessagingException e) {
e.printStackTrace();
}
}
private void handleMessage(Message msg) throws MessagingException, IOException {
var cType = msg.getContentType();
if (cType.contains(MediaType.TEXT_PLAIN_VALUE)) {
handleText((String) msg.getContent());
} else if (cType.contains(MediaType.MULTIPART_MIXED_VALUE)) {
handleMultipart((Multipart) msg.getContent());
}
}
// See
// https://stackoverflow.com/questions/1748183/download-attachments-using-java-mail
private void handleMultipart(Multipart msgContent) throws MessagingException, IOException {
var mCount = msgContent.getCount();
for (var i = 0; i < mCount; i++) {
this.processAttachments(msgContent.getBodyPart(i));
}
}
private void processAttachments(BodyPart part) throws IOException, MessagingException {
var content = part.getContent();
if (content instanceof InputStream || content instanceof String) {
if (Part.ATTACHMENT.equalsIgnoreCase(part.getDisposition()) || !part.getFileName().isBlank()) {
var fName = String.format("%s.%s", UUID.randomUUID().toString(),
FilenameUtils.getExtension(part.getFileName()));
FileUtils.copyInputStreamToFile(part.getInputStream(), new File(outDir + File.separator + fName));
}
if (content instanceof Multipart) {
Multipart multipart = (Multipart) content;
for (int i = 0; i < multipart.getCount(); i++) {
var bodyPart = multipart.getBodyPart(i);
processAttachments(bodyPart);
}
}
}
}
}
Whenever I run my code using the config above, I receive the following error:
javax.mail.MessagingException: No inputstream from datasource;
nested exception is:
java.lang.IllegalStateException: Folder is not Open
at javax.mail.internet.MimeMultipart.parse(MimeMultipart.java:576)
at javax.mail.internet.MimeMultipart.getCount(MimeMultipart.java:312)
at com.midamcorp.data.mail.MailReceivingMessageHandler.handleMultipart(MailReceivingMessageHandler.java:70)
at com.midamcorp.data.mail.MailReceivingMessageHandler.handleMessage(MailReceivingMessageHandler.java:58)
at com.midamcorp.data.mail.MailReceivingMessageHandler.handleMessageInternal(MailReceivingMessageHandler.java:44)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:62)
at org.springframework.integration.handler.ReplyProducingMessageHandlerWrapper.handleRequestMessage(ReplyProducingMessageHandlerWrapper.java:58)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:134)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:62)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:570)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:520)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.handleMessage(SourcePollingChannelAdapter.java:196)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.messageReceived(AbstractPollingEndpoint.java:444)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:428)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:376)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$null$3(AbstractPollingEndpoint.java:323)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.lambda$execute$0(ErrorHandlingTaskExecutor.java:57)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:55)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$4(AbstractPollingEndpoint.java:320)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Folder is not Open
at com.sun.mail.pop3.POP3Folder.checkOpen(POP3Folder.java:562)
at com.sun.mail.pop3.POP3Folder.getProtocol(POP3Folder.java:592)
at com.sun.mail.pop3.POP3Message.getRawStream(POP3Message.java:154)
at com.sun.mail.pop3.POP3Message.getContentStream(POP3Message.java:251)
at javax.mail.internet.MimePartDataSource.getInputStream(MimePartDataSource.java:78)
at javax.mail.internet.MimeMultipart.parse(MimeMultipart.java:570)
... 35 more
Obviously, the root cause is clear - the POP3 folder is closed. I have seen solutions that would likely be able to handle when just the Java mail classes are used, but none with Spring Integration. My question is how does one properly control when a folder is open or closed using Spring Integration Mail? I realize the Pop3MailReceiver class has a .setAutoCloseFolder() method. Based on the Spring Docs, I assume I need to set that, along something like the following to my handler:
Closeable closeableResource = StaticMessageHeaderAccessor.getCloseableResource(message);
if (closeableResource != null) {
closeableResource.close();
}
However, if I set autoCloseFolder to false, it does not appear as if the message even ever "hits" my handler, so unfortunately being able to close the resource does not even matter at this point. That is, when autoClose is set to false, the 'handleMessageInternal()' method in my handler class is never reached even though there are indeed message on the POP3 server. Instead I just get a bunch of logs like this:
2020-06-26 15:26:54.523 INFO 15348 --- [ scheduling-1] o.s.integration.mail.Pop3MailReceiver : attempting to receive mail from folder [INBOX]
What am I missing?
Thanks.

Can't poll from S3 with a custom endpoint configuration

I try to use spring-integration-aws to poll from a S3 bucket to trigger a spring-batch job. My S3 bucket is not hosted on Amazon, it's on a local minio server, so I got a custom configuration :
#Bean
public AmazonS3 amazonS3(ConfigProperties configProperties) {
return AmazonS3ClientBuilder.standard()
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://localhost:9001","eu-west-0")) // Region matches with minio region
.withPathStyleAccessEnabled(configProperties.getS3().isPathStyle())
.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(
configProperties.getS3().getAccessKey(), configProperties.getS3().getSecretKey()
))).build();
}
I defined my IntegrationFlow this way :
#Bean
public IntegrationFlow s3InboundFlow() {
S3RemoteFileTemplate template = new S3RemoteFileTemplate(new S3SessionFactory(amazonS3));
S3StreamingMessageSource s3StreamingMessageSource = new S3StreamingMessageSource(template);
s3StreamingMessageSource.setRemoteDirectory(String.format("%s/OUT/", configProperties.getS3().getBucketDataPath()));
return IntegrationFlows.from(s3StreamingMessageSource, configurer -> configurer
.id("s3InboundAdapter")
.autoStartup(true)
.poller(Pollers.fixedDelay(POLL, TimeUnit.SECONDS)))
.handle(jobLaunchingGateway(jobRepository)) // Launch a spring-batch job
.get();
}
The problem is, when the polling occured, I got the following error :
2020-03-30 19:05:21,008 ERROR [scheduling-1] org.springframework.integration.handler.LoggingHandler: org.springframework.messaging.MessagingException: nested exception is java.lang.IllegalStateException: S3 client with invalid S3 endpoint configured: localhost:9001
at org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:342)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$null$1(AbstractPollingEndpoint.java:275)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.lambda$execute$0(ErrorHandlingTaskExecutor.java:57)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:55)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$2(AbstractPollingEndpoint.java:272)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: S3 client with invalid S3 endpoint configured: localhost:9001
at com.amazonaws.services.s3.AmazonS3Client.getRegion(AmazonS3Client.java:4270)
at org.springframework.integration.aws.support.S3Session.getHostPort(S3Session.java:228)
at org.springframework.integration.file.remote.AbstractRemoteFileStreamingMessageSource.doReceive(AbstractRemoteFileStreamingMessageSource.java:214)
at org.springframework.integration.endpoint.AbstractMessageSource.receive(AbstractMessageSource.java:167)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:250)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:359)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:328)
It happens because when the file is received, some headers are set in spring-integration-aws :
AbstractRemoteFileStreamingMessageSource.java
return getMessageBuilderFactory()
.withPayload(session.readRaw(remotePath))
.setHeader(IntegrationMessageHeaderAccessor.CLOSEABLE_RESOURCE, session)
.setHeader(FileHeaders.REMOTE_DIRECTORY, file.getRemoteDirectory())
.setHeader(FileHeaders.REMOTE_FILE, file.getFilename())
.setHeader(FileHeaders.REMOTE_HOST_PORT, session.getHostPort())
.setHeader(FileHeaders.REMOTE_FILE_INFO,
this.fileInfoJson ? file.toJson() : file);
The REMOTE_HOST_PORT header is set thanks to getHostPort() method. The getHostPort() in S3Session.java then calls the getRegion() method.
The getRegion() method in AmazonS3Client is not checking if a value is set by the user in signing region field. It only checks if the host is matching the "amazonaws.com" pattern.
#Override
public String getHostPort() {
Region region = this.amazonS3.getRegion().toAWSRegion();
return String.format("%s.%s.%s:%d", AmazonS3.ENDPOINT_PREFIX, region.getName(), region.getDomain(), 443);
}
#Override
public synchronized Region getRegion() {
String authority = super.endpoint.getAuthority();
if (Constants.S3_HOSTNAME.equals(authority)) {
return Region.US_Standard;
} else {
Matcher m = Region.S3_REGIONAL_ENDPOINT_PATTERN.matcher(authority);
if (m.matches()) {
return Region.fromValue(m.group(1));
} else {
throw new IllegalStateException(
"S3 client with invalid S3 endpoint configured: " + authority);
}
}
}
How is it possible to poll from S3 with a custom endpoint configuration ?
Why is the getHostPort() method not checking the signing region value ? Is it possible to workaround this ?
Yes, it is possible to workaround.
You just extend an S3SessionFactory to return an extension of S3Session with overridden getHostPort() method for your custom endpoint.
public class MyS3SessionFactory extends S3SessionFactory {
private MyS3Session s3Session;
#Override
public S3Session getSession() {
return s3Session;
}
public MyS3SessionFactory(AmazonS3 amazonS3) {
super(amazonS3);
Assert.notNull(amazonS3, "'amazonS3' must not be null.");
this.s3Session = new MyS3Session(amazonS3);
}
}
public class MyS3Session extends S3Session {
public MyS3Session(AmazonS3 amazonS3) {
super(amazonS3);
}
#Override
public String getHostPort() {
return "";
}
}
Let's discuss a possible fix in the issue you have raised: https://github.com/spring-projects/spring-integration-aws/issues/160 !

My .onSucceed (okhttp) throws "Service must only be used fromm FX Application Thread"

I wrote an Interceptor which catches the 401 HTTP Status and then calls a method which sends a RefreshToken to the server to get a new AccessToken. For some reason it doesn't work although in the same class I have got a second function which does almost the same and works completely fine.
Am i missing out on something?
Thanks in advance.
Function where error occures:
public void RefreshToken(){
try{
if(db==null){
db=Database.newInstance();
}
}catch(Exception ex){
System.out.println("Error in while getting new Database");
}
Service<String> t=db.GetAccessToken();
t.setOnSucceeded(e -> { //Error thrown here
Database.ACCESSTOKEN=t.getValue();
});
t.setOnFailed(e -> {
System.out.println("Error with HTTP GET new Token");
});
}
Almost the same Function:
private void Login(String username, String password) {
Service<BothTokens> t=db.Login(new Credentials(username,password));
t.setOnSucceeded(e -> {
Database.REFRESHTOKEN=t.getValue().getRefreshToken();
Database.ACCESSTOKEN=t.getValue().getAccessToken();
openMainFXML();
closeWindow();
});
t.setOnFailed(e -> {
new ServiceExceptionHandler<>(t).handleServiceWithRetryOrClose((Stage) btnLogin.getScene().getWindow(),
"Error beim Login",
t.getException().getMessage(), (Exception) t.getException());
});
}
Interceptor:
OkHttpClient OK_HTTP_CLIENT = new OkHttpClient.Builder().addInterceptor(chain -> {
Response originalResponse = chain.proceed(chain.request());
if(originalResponse.code()==401){
try{
Login_Controller login=Login_Controller.newInstance();
login.RefreshToken();
}catch(Exception ex){
System.out.println("Error getting new Token");
ex.printStackTrace();
}
}
return originalResponse;
}).build();
Exception:
java.lang.IllegalStateException: Service must only be used from the FX Application Thread
at javafx.concurrent.Service.checkThread(Service.java:906)
at javafx.concurrent.Service.setOnSucceeded(Service.java:409)
at pkgController.Login_Controller.RefreshToken(Login_Controller.java:106)
at pkgServices.statics.StaticServerObjects.lambda$static$0(StaticServerObjects.java:19)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.kt:184)
at okhttp3.RealCall.execute(RealCall.kt:66)
at pkgServices.buildings.GetBuildings$1.call(GetBuildings.java:32)
at pkgServices.buildings.GetBuildings$1.call(GetBuildings.java:25)
at javafx.concurrent.Task$TaskCallable.call(Task.java:1423)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at javafx.concurrent.Service.lambda$null$493(Service.java:725)
at java.security.AccessController.doPrivileged(Native Method)
at javafx.concurrent.Service.lambda$executeTask$494(Service.java:724)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
java.lang.Exception: java.lang.Exception: Exception occured in GetBuildings with code: 401
at pkgServices.buildings.GetBuildings$1.call(GetBuildings.java:40)
at pkgServices.buildings.GetBuildings$1.call(GetBuildings.java:25)
at javafx.concurrent.Task$TaskCallable.call(Task.java:1423)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at javafx.concurrent.Service.lambda$null$493(Service.java:725)
at java.security.AccessController.doPrivileged(Native Method)
at javafx.concurrent.Service.lambda$executeTask$494(Service.java:724)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Exception occured in GetBuildings with code: 401
at pkgServices.buildings.GetBuildings$1.call(GetBuildings.java:37)
... 9 more
i suggest javafx AsyncTask Libray
and prepare your requests as below:
public class Example extends AsyncTask<Void,Void,String> {
private UIController controller;
public Example(UIController controller) {
this.controller = controller;
}
#Override
void onPreExecute() {
//PREPARE GUI, SHOW PROGRESS OR SOMETHING....
this.controller.updateProgressLabel("Starting Download")
}
#Override
String doInBackground() {
//THIS METHOD WORKING IN BACKGROUND HERE YOU SOULD IMPLEMENT CODE OF HTTP REQUEST AND RETURN RESULT OF REQUEST FOR EXAMPLE OBJECT OR JSON STRING
return new String();
}
#Override
void onPostExecute(String result) {
//HERE YOU CAN GET YOUR DATA FROM REQUEST AND SET RESULT ON GUI
this.controller.updateProgressLabel(result);
}
#Override
void progressCallback(Object... params) {
}
}
The problem was that the function was running in a different thread than the FXML-Controller and I had to write
Platform.runLater(() -> {....} )

Trader-Demo Flow tests error : KotlinNullPointerException

My question is how can i upload the attachment in flow test to make this test successful.
This is the test that im doing in trader-demo flowtest
private fun issue_commercial_paper( amount: Amount<Currency>,
issueRef: OpaqueBytes,
recipient: Party,
notary: Party,
progressTracker: ProgressTracker): SignedTransaction {
val p = CommercialPaperIssueFlow(amount,issueRef,recipient,notary,progressTracker)
val future = BankCorpNode.services.startFlow(p)
mockNet.runNetwork()
return future.resultFuture.getOrThrow()
}
#Test
fun `trade issuance`(){
issue_commercial_paper(1000.DOLLARS,OpaqueBytes.of(42.toByte()),ACorp,notary,CommercialPaperIssueFlow.tracker())
val pass = SellerFlow(BCorp ,500.DOLLARS, SellerFlow.tracker() )
val future = ACorpNode.services.startFlow(pass)
mockNet.runNetwork()
val results = future.resultFuture.getOrThrow()
assertNotNull(results)
}
This is the error
kotlin.KotlinNullPointerException
at net.corda.traderdemo.flow.CommercialPaperIssueFlow.call(CommercialPaperIssueFlow.kt:54)
at net.corda.traderdemo.flow.CommercialPaperIssueFlow.call(CommercialPaperIssueFlow.kt:25)
at net.corda.node.services.statemachine.FlowStateMachineImpl.run(FlowStateMachineImpl.kt:96)
at net.corda.node.services.statemachine.FlowStateMachineImpl.run(FlowStateMachineImpl.kt:44)
at co.paralleluniverse.fibers.Fiber.run1(Fiber.java:1092)
at co.paralleluniverse.fibers.Fiber.exec(Fiber.java:788)
at co.paralleluniverse.fibers.RunnableFiberTask.doExec(RunnableFiberTask.java:100)
at co.paralleluniverse.fibers.RunnableFiberTask.run(RunnableFiberTask.java:91)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at net.corda.node.utilities.AffinityExecutor$ServiceAffinityExecutor$1$thread$1.run(AffinityExecutor.kt:63)
you can refer to this sample to see if you can help me guys thanks in advance
https://github.com/corda/corda/tree/c4ceca378762fe1959f075a1c8b1c301e411b6b8/samples/trader-demo
Suppose your attachment is under main/resources/my-attachment.jar. Within a Java flow test, you upload this attachment to the node as follows:
nodeA.transaction(() -> {
InputStream attachmentStream = getClass().getClassLoader().getResourceAsStream("my-attachment.jar");
try {
nodeA.getServices().getAttachments().importAttachment(attachmentStream, "joel", "myAttachment");
} catch (IOException e) {
e.printStackTrace();
}
// The attachment is now uploaded to the node.
// TODO: Remainder of the test.
return null;
});
Or in Kotlin:
nodeA.transaction {
val attachmentStream = javaClass.classLoader.getResourceAsStream("my-attachment.jar")
nodeA.services.attachments.importAttachment(attachmentStream, "joel", "myAttachment")
// The attachment is now uploaded to the node.
// TODO: Remainder of the test.
}

Categories