I'm starting a new project with 'spring-cloud-dataflow', developing a bunch of jar to fit my need.
One of this is a processor to untar files coming from a file source, this application use a customized version of integration-zip with features to handle tar and gunzip file compression.
So my problem is the following one: while my source send a single message with the file reference the processor receive those message multiple time, same payload but with different id.
Here the log file of both component
As you can see file produce only on message:
2017-10-02 12:38:28.013 INFO 17615 --- [ask-scheduler-3] o.s.i.file.FileReadingMessageSource : Created message: [GenericMessage [payload=/tmp/patent/CNINO_im_201733_batch108.tgz, headers={id=0b99b840-e3b3-f742-44ec-707aeea638c8, timestamp=1506940708013}]]
while producer has 3 message incoming:
2017-10-02 12:38:28.077 INFO 17591 --- [ -L-1] o.s.i.codec.kryo.CompositeKryoRegistrar : registering [40, java.io.File] with serializer org.springframework.integration.codec.kryo.FileSerializer
2017-10-02 12:38:28.080 INFO 17591 --- [ -L-1] .w.c.s.a.c.p.AbstractCompressTransformer : Message 'GenericMessage [payload=/tmp/patent/CNINO_im_201733_batch108.tgz, headers={kafka_offset=1, id=1
a4d4b9c-86fe-d3a8-d800-8013e8ae7027, kafka_receivedPartitionId=0, contentType=application/x-java-object;type=java.io.File, kafka_receivedTopic=untar.file, timestamp=1506940708079}]' unpacking started...
2017-10-02 12:38:28.080 INFO 17591 --- [ -L-1] .w.c.s.a.c.p.AbstractCompressTransformer : Check message's payload type to decompress
2017-10-02 12:38:29.106 INFO 17591 --- [ -L-1] .w.c.s.a.c.p.AbstractCompressTransformer : Message 'GenericMessage [payload=/tmp/patent/CNINO_im_201733_batch108.tgz, headers={kafka_offset=1, id=c
d611ca4-4cd9-0624-0871-dcf93a9a0051, kafka_receivedPartitionId=0, contentType=application/x-java-object;type=java.io.File, kafka_receivedTopic=untar.file, timestamp=1506940709106}]' unpacking started...
2017-10-02 12:38:29.107 INFO 17591 --- [ -L-1] .w.c.s.a.c.p.AbstractCompressTransformer : Check message's payload type to decompress
2017-10-02 12:38:31.108 INFO 17591 --- [ -L-1] .w.c.s.a.c.p.AbstractCompressTransformer : Message 'GenericMessage [payload=/tmp/patent/CNINO_im_201733_batch108.tgz, headers={kafka_offset=1, id=97171a2e-29ac-2111-b838-3da7220f5e3c, kafka_receivedPartitionId=0, contentType=application/x-java-object;type=java.io.File, kafka_receivedTopic=untar.file, timestamp=1506940711108}]' unpacking started...
2017-10-02 12:38:31.108 INFO 17591 --- [ -L-1] .w.c.s.a.c.p.AbstractCompressTransformer : Check message's payload type to decompress
2017-10-02 12:38:31.116 ERROR 17591 --- [ -L-1] o.s.integration.handler.LoggingHandler : org.springframework.integration.transformer.MessageTransformationException: failed to transform message; nested exception is org.springframework.messaging.MessageHandlingException: Failed to apply Zip transformation.; nested exception is java.io.FileNotFoundException: /tmp/patent/CNINO_im_201733_batch108.tgz (File o directory non esistente), failedMessage=GenericMessage [payload=/tmp/patent/CNINO_im_201733_batch108.tgz, headers={kafka_offset=1, id=97171a2e-29ac-2111-b838-3da7220f5e3c, kafka_receivedPartitionId=0, contentType=application/x-java-object;type=java.io.File, kafka_receivedTopic=untar.file, timestamp=1506940711108}], failedMessage=GenericMessage [payload=/tmp/patent/CNINO_im_201733_batch108.tgz, headers={kafka_offset=1, id=97171a2e-29ac-2111-b838-3da7220f5e3c, kafka_receivedPartitionId=0, contentType=application/x-java-object;type=java.io.File, kafka_receivedTopic=untar.file, timestamp=1506940711108}]
at org.springframework.integration.transformer.AbstractTransformer.transform(AbstractTransformer.java:44)
I can't find any solution to this problem, does anybody has the same problem and found a way to fix it? Or there's any configuration I miss?
EDIT:
I'm using the local version of SDFS version 1.2.2.RELEASE, so IO file operation work on the same filesystem, and I use version Ditmars.BUILD-SNAPSHOT for SCS.
Unfortunatly if I disable the file delete operation application, this app still process message multiple time. Here some code snippet, and i you like this is my project repo:
This is my processor class:
#EnableBinding(Processor.class)
#EnableConfigurationProperties(UnTarProperties.class)
public class UnTarProcessor {
#Autowired
private UnTarProperties properties;
#Autowired
private Processor processor;
#Bean
public UncompressedResultSplitter splitter() {
return new UncompressedResultSplitter();
}
#Bean
public UnTarGzTransformer transformer() {
UnTarGzTransformer unTarGzTransformer = new UnTarGzTransformer(properties.isUseGzCompression());
unTarGzTransformer.setExpectSingleResult(properties.isSingleResult());
unTarGzTransformer.setWorkDirectory(new File(properties.getWorkDirectory()));
unTarGzTransformer.setDeleteFiles(properties.isDeleteFile());
return unTarGzTransformer;
}
#Bean
public IntegrationFlow process() {
return IntegrationFlows.from(processor.input())
.transform(transformer())
.split(splitter())
.channel(processor.output())
.get();
}
}
This is the core method used to decompress file:
#Override
protected Object doCompressTransform(final Message<?> message) throws Exception {
logger.info(String.format("Message '%s' unpacking started...", message));
try (InputStream checkMessage = checkMessage(message);
InputStream inputStream = (gzCompression ? new BufferedInputStream(new GZIPInputStream(checkMessage)) : new BufferedInputStream(checkMessage))) {
final Object payload = message.getPayload();
final Object unzippedData;
try (TarArchiveInputStream tarIn = new TarArchiveInputStream(inputStream)){
TarArchiveEntry entry = null;
final SortedMap<String, Object> uncompressedData = new TreeMap<String, Object>();
while ((entry = (TarArchiveEntry) tarIn.getNextEntry()) != null) {
final String zipEntryName = entry.getName();
final Date zipEntryTime = entry.getLastModifiedDate();
final long zipEntryCompressedSize = entry.getSize();
final String type = entry.isDirectory() ? "directory" : "file";
final File tempDir = new File(workDirectory, message.getHeaders().getId().toString());
tempDir.mkdirs(); // NOSONAR false positive
final File destinationFile = new File(tempDir, zipEntryName);
if (entry.isDirectory()) {
destinationFile.mkdirs(); // NOSONAR false positive
}
else {
unpackEntries(tarIn, entry, tempDir);
uncompressedData.put(zipEntryName, destinationFile);
}
}
if (uncompressedData.isEmpty()) {
unzippedData = null;
}
else {
if (this.expectSingleResult) {
if (uncompressedData.size() == 1) {
unzippedData = uncompressedData.values().iterator().next();
}
else {
throw new MessagingException(message, String.format("The UnZip operation extracted %s "
+ "result objects but expectSingleResult was 'true'.", uncompressedData.size()));
}
}
else {
unzippedData = uncompressedData;
}
}
logger.info("Payload unpacking completed...");
}
finally {
if (payload instanceof File && this.deleteFiles) {
final File filePayload = (File) payload;
if (!filePayload.delete() && logger.isWarnEnabled()) {
if (logger.isWarnEnabled()) {
logger.warn("failed to delete File '" + filePayload + "'");
}
}
}
}
return unzippedData;
}
catch (Exception e) {
throw new MessageHandlingException(message, "Failed to apply Zip transformation.", e);
}
}
Exception is thrown by method checkmessage()
protected InputStream checkMessage(Message<?> message) throws FileNotFoundException {
logger.info("Check message's payload type to decompress");
InputStream inputStream;
Object payload = message.getPayload();
if (payload instanceof File) {
final File filePayload = (File) payload;
if (filePayload.isDirectory()) {
throw new UnsupportedOperationException(String.format("Cannot unzip a directory: '%s'",
filePayload.getAbsolutePath()));
}
inputStream = new FileInputStream(filePayload);
}
else if (payload instanceof InputStream) {
inputStream = (InputStream) payload;
}
else if (payload instanceof byte[]) {
inputStream = new ByteArrayInputStream((byte[]) payload);
}
else {
throw new IllegalArgumentException(String.format("Unsupported payload type '%s'. " +
"The only supported payload types are java.io.File, byte[] and java.io.InputStream",
payload.getClass().getSimpleName()));
}
return inputStream;
}
I really appreciate any help.
Thanks a lot
We would need more information. Version of SCDF and SCS apps. Your DSL on how you deployed your apps at very least.
Just checked your logs, did you realize your consumer is failing to consume the message due a FileNotFoundException? You are not receiving the same message multiple times, SCS is just trying to redeliver it before failing. Check your full logs and how you are failing to open the file in the specified location
The exception is happening at your transformer, you receive the message multiple times due retry configuration of SCS, since the error is in your logic, is hard to follow. It says FileNotFoundException I don't know what in your process puts the file there, that may be the reason. It does not appear to be anything with SCS
Related
Java Code
#Component
#RequiredArgsConstructor
public class WireInboundFileListener extends RouteBuilder {
private final JwtWireTokenService jwtWireTokenService;
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
#Value("${wire.ftp.protocol}")
private String ftpProtocol;
#Value("${wire.ftp.server}")
private String ftpServer;
#Value("${wire.ftp.server.port}")
private String ftpServerPort;
#Value("${wire.ftp.username}")
private String ftpUsername;
#Value("${wire.ftp.password}")
private String ftpPassword;
#Value("${wire.ftp.private.key.file}")
private String ftpPrivateKeyFile;
#Value("${wire.ftp.private.key.passphrase}")
private String privateKeyPassphrase;
#Value("${wire.file.inbound.dir}")
private String ftpListenDir;
#Value("${wire.file.inbound.url}")
private String inboundUrl;
#Override
public void configure() {
var fromFtpUri = String.format("%s:%s:%s/%s?username=%s&delete=true&antInclude=*.txt",
ftpProtocol, ftpServer, ftpServerPort, ftpListenDir, ftpUsername);
log.info("SFTP inbound listen dir : " + ftpListenDir);
if (Environment.getExecutionEnvironment().equals(Environment.ExecutionEnvironment.AWS)) {
fromFtpUri += "&privateKeyFile=" + ftpPrivateKeyFile + "&privateKeyPassphrase=" + privateKeyPassphrase;
} else {
fromFtpUri += "&password=" + ftpPassword;
}
from(fromFtpUri)
//.delay(10000) event I put delay but still got file content null
.convertBodyTo(String.class)
.process(exchange -> {
final var requestBody = new HashMap<String, Object>();
final var content = exchange.getIn().getBody();
final var fileName = exchange.getIn().getHeader("CamelFileName");
requestBody.put("content", content);
requestBody.put("name", fileName);
exchange.getIn().setBody(OBJECT_MAPPER.writeValueAsString(requestBody));
})
.to("log:com.test.wire.inbound.listener.SftpRoute")
.setHeader(Exchange.HTTP_METHOD, constant("POST"))
.setHeader("Content-Type", constant("application/json"))
.setHeader("Authorization", method(this, "clientToken"))
.to(inboundUrl + "?throwExceptionOnFailure=false")
.log("Response body from wire inbound : ${body}")
.end();
}
public String clientToken() {
return "Bearer " + jwtWireTokenService.getToken();
}
}
Success Request
2022-04-20 03:46:47.910 INFO 1 --- [read #6 - Delay] c.test.wire.inbound.listener.SftpRoute : Exchange[ExchangePattern: InOnly, BodyType: String, Body: {
"name": "sample-inbound-transfer-20220414024722.txt",
"content": "file content"
}]
Fail Request
2022-04-21 09:36:54.148 INFO 1 --- [read #4 - Delay] c.test.wire.inbound.listener.SftpRoute : Exchange[ExchangePattern: InOnly, BodyType: String, Body: {
"name": "sample-inbound-transfer-20220414024722.txt",
"content": ""
}]
Main issue
final var content = exchange.getIn().getBody();// sometimes get null, sometimes can get file contents
When I test to drop a file to SFTP server in local it worked as I expected in Sucess Request because the process to upload seemed fast because it is in local (FileZilla).
But when I test to drop a file again to SFTP server that hosts in real server sometimes it works and sometimes it didn't work Fail Request. Seem like the SFTP consuming file issue. Could you please help me on that? Thanks
The file is probably (sometimes) consumed by your Camel route before the file upload is finished.
What you can try is to configure your endpoint to poll the files only if it has exclusive read-lock on the file (i.e. the file is not in-progress or being written); see the readLock parameter.
I think the default is no read lock; you should set readLock=changed...but I do not remember is such mode is also possible on endpoints of SFTP type.
Help me please if you can)
I am trying to use google speech to text API. I have uploaded an audio file ( record1638730550724.wav) on google storage bucket with the name "summaryapp". But when i try to make longrecognize request i get this error
2021-12-05 18:56:20 INFO RecordController:60 -
com.google.api.gax.rpc.InvalidArgumentException:
io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Audio URI
gc://summaryapp/record1638730255219.wav is an invalid GCS path.
RecognizeRequestHandler class:
public class RecognizeRequestHandler {
private static CallsInfo callsInfo;
#Autowired
private static CallsInfoDaoImpl callsInfoDao;
public static void asyncRecognizeGcs(String gcsUri, int oid) throws Exception {
// Configure polling algorithm
SpeechSettings.Builder speechSettings = SpeechSettings.newBuilder();
TimedRetryAlgorithm timedRetryAlgorithm =
OperationTimedPollAlgorithm.create(
RetrySettings.newBuilder()
.setInitialRetryDelay(Duration.ofMillis(500L))
.setRetryDelayMultiplier(1.5)
.setMaxRetryDelay(Duration.ofMillis(5000L))
.setInitialRpcTimeout(Duration.ZERO) // ignored
.setRpcTimeoutMultiplier(1.0) // ignored
.setMaxRpcTimeout(Duration.ZERO) // ignored
.setTotalTimeout(Duration.ofHours(24L)) // set polling timeout to 24 hours
.build());
speechSettings.longRunningRecognizeOperationSettings().setPollingAlgorithm(timedRetryAlgorithm);
// Instantiates a client with GOOGLE_APPLICATION_CREDENTIALS
try (SpeechClient speech = SpeechClient.create(speechSettings.build())) {
// Configure remote file request for FLAC
RecognitionConfig config =
RecognitionConfig.newBuilder()
.setEncoding(AudioEncoding.LINEAR16)
.setLanguageCode("ru-RU")
.setSampleRateHertz(16000)
.build();
RecognitionAudio audio = RecognitionAudio.newBuilder().setUri(gcsUri).build();
// Use non-blocking call for getting file transcription
OperationFuture<LongRunningRecognizeResponse, LongRunningRecognizeMetadata> response =
speech.longRunningRecognizeAsync(config, audio);
while (!response.isDone()) {
System.out.println("Waiting for response...");
Thread.sleep(10000);
}
List<SpeechRecognitionResult> results = response.get().getResultsList();
for (SpeechRecognitionResult result : results) {
// There can be several alternative transcripts for a given chunk of speech. Just use the
// first (most likely) one here.
SpeechRecognitionAlternative alternative = result.getAlternativesList().get(0);
System.out.printf("Transcription: %s\n", alternative.getTranscript());
//Trying to write response to database
Date date = new Date(System.currentTimeMillis());
String text = alternative.getTranscript();
callsInfo = new CallsInfo(oid, date, text);
callsInfoDao.create(callsInfo);
}
}
}
}
And this is RecordController class, which call Recognize class with GCS URL
public class RecordController {
public static Logger logger = LoggerFactory.getLogger(RecordController.class);
#Autowired
AuthToken authToken;
#Autowired
AuthTokenDaoImpl authTokenDao;
#Autowired
AudioRecord audioRecord;
#Autowired
RecordDaoImpl recordDao;
Thread t;
private String RESULT = "NO MATCHING";
private String GCS_URL = "gc://summaryapp/";
private String RECORD_FILE_NAME = "";
#PostMapping(value = "/newAudioRecord", consumes = "application/json",
produces = "text/plain;charset=UTF-8")
public String postNewRecord(#RequestParam(value = "token", required = true) String token,
#RequestBody AudioRecord audioRecord) throws JsonProcessingException {
System.out.println(audioRecord.getOid() + " " + audioRecord.getRecordFileName());
authToken = authTokenDao.readByToken(token);
if (authToken != null) {
RECORD_FILE_NAME = audioRecord.getRecordFileName();
audioRecord.setOid(authToken.getOid());
recordDao.create(audioRecord);
AudioRecord resultAudioRecord = recordDao.readByName(audioRecord.getOid(),
audioRecord.getRecordFileName());
ObjectMapper mapper = new ObjectMapper();
RESULT = mapper.writeValueAsString(resultAudioRecord);
try {
RecognizeRequestHandler.asyncRecognizeGcs(GCS_URL+RECORD_FILE_NAME, audioRecord.getOid());
} catch (Exception e) {
logger.info(e.getMessage());
}
}
return RESULT;
}
}
UPD: After changing gc:// to gs:// in path i am do not get any messages about "invalid path". But now process ends with following lines in logs:
2021-12-06 18:37:34 INFO RecognizeRequestHandler:108 - -------->
Result list size: 0 2021-12-06 18:37:34 DEBUG NettyClientHandler:214 -
[id: 0xa576e0da, L:/172.18.0.2:49394 -
R:speech.googleapis.com/64.233.164.95:443] OUTBOUND GO_AWAY:
lastStreamId=0 errorCode=0 length=0 bytes= 2021-12-06 18:37:35 DEBUG
PoolThreadCache:229 - Freed 24 thread-local buffer(s) from thread:
grpc-default-worker-ELG-1-4
As seen from "sout debug" - line of code
List<SpeechRecognitionResult> results = response.get().getResultsList();
returns a result list size: 0
--------> Result list size: 0
UPD: Audio recorded with next settings:
int mic = MediaRecorder.AudioSource.MIC;
recorder.setAudioSource(mic);
recorder.setOutputFormat(output_formats[currentFormat]); //MediaRecorder.OutputFormat.AAC_ADTS
recorder.setAudioEncoder(encoder); //MediaRecorder.AudioEncoder.AAC;
recorder.setAudioSamplingRate(sampleRate); //16000
recorder.setOutputFile(CurrentAudioFilePath);
recorder.setOnErrorListener(errorListener);
recorder.setOnInfoListener(infoListener);
I have a Spring Boot project that is leveraging Spring Integration. My goal is to to poll a POP3 mail server regularly, and download any attachments associated with those messages. My relevant Spring Config looks like this:
#Configuration
public class MailIntegrationConfig {
#Value("${path.output.temp}")
private String outPath;
#Bean
public MessageChannel mailChannel() {
return new DirectChannel();
}
#Bean
#InboundChannelAdapter(value = "mailChannel", poller = #Poller(fixedDelay = "16000"))
public MessageSource<Object> fileReadingMessageSource() {
var receiver = new Pop3MailReceiver("pop3s://user:passwordexample.com/INBOX");
var mailProperties = new Properties();
mailProperties.setProperty("mail.pop3.port", "995");
mailProperties.put("mail.pop3.ssl.enable", true);
receiver.setShouldDeleteMessages(false);
receiver.setMaxFetchSize(10);
receiver.setJavaMailProperties(mailProperties);
// receiver.setHeaderMapper(new DefaultMailHeaderMapper());
var source = new MailReceivingMessageSource(receiver);
return source;
}
#Bean
#ServiceActivator(inputChannel = "mailChannel")
public MessageHandler popMessageHandler() {
return new MailReceivingMessageHandler(outPath);
}
}
My MailReceivingMessageHandler class (partial)
public class MailReceivingMessageHandler extends AbstractMessageHandler {
private String outDir;
public MailReceivingMessageHandler(String outDir) {
var outPath = new File(outDir);
if (!outPath.exists()) {
throw new IllegalArgumentException(String.format("%s does not exist.", outDir));
}
this.outDir = outDir;
}
#Override
protected void handleMessageInternal(org.springframework.messaging.Message<?> message) {
Object payload = message.getPayload();
if (!(payload instanceof Message)) {
throw new IllegalArgumentException(
"Unable to create MailMessage from payload type [" + message.getPayload().getClass().getName()
+ "], " + "expected MimeMessage, MailMessage, byte array or String.");
}
try {
var msg = (Message) payload;
System.out.println(String.format("Headers [%s] Subject [%s]. Content-Type [%s].", msg.getAllHeaders(),
msg.getSubject(), msg.getContentType()));
this.handleMessage(msg);
} catch (IOException | MessagingException e) {
e.printStackTrace();
}
}
private void handleMessage(Message msg) throws MessagingException, IOException {
var cType = msg.getContentType();
if (cType.contains(MediaType.TEXT_PLAIN_VALUE)) {
handleText((String) msg.getContent());
} else if (cType.contains(MediaType.MULTIPART_MIXED_VALUE)) {
handleMultipart((Multipart) msg.getContent());
}
}
// See
// https://stackoverflow.com/questions/1748183/download-attachments-using-java-mail
private void handleMultipart(Multipart msgContent) throws MessagingException, IOException {
var mCount = msgContent.getCount();
for (var i = 0; i < mCount; i++) {
this.processAttachments(msgContent.getBodyPart(i));
}
}
private void processAttachments(BodyPart part) throws IOException, MessagingException {
var content = part.getContent();
if (content instanceof InputStream || content instanceof String) {
if (Part.ATTACHMENT.equalsIgnoreCase(part.getDisposition()) || !part.getFileName().isBlank()) {
var fName = String.format("%s.%s", UUID.randomUUID().toString(),
FilenameUtils.getExtension(part.getFileName()));
FileUtils.copyInputStreamToFile(part.getInputStream(), new File(outDir + File.separator + fName));
}
if (content instanceof Multipart) {
Multipart multipart = (Multipart) content;
for (int i = 0; i < multipart.getCount(); i++) {
var bodyPart = multipart.getBodyPart(i);
processAttachments(bodyPart);
}
}
}
}
}
Whenever I run my code using the config above, I receive the following error:
javax.mail.MessagingException: No inputstream from datasource;
nested exception is:
java.lang.IllegalStateException: Folder is not Open
at javax.mail.internet.MimeMultipart.parse(MimeMultipart.java:576)
at javax.mail.internet.MimeMultipart.getCount(MimeMultipart.java:312)
at com.midamcorp.data.mail.MailReceivingMessageHandler.handleMultipart(MailReceivingMessageHandler.java:70)
at com.midamcorp.data.mail.MailReceivingMessageHandler.handleMessage(MailReceivingMessageHandler.java:58)
at com.midamcorp.data.mail.MailReceivingMessageHandler.handleMessageInternal(MailReceivingMessageHandler.java:44)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:62)
at org.springframework.integration.handler.ReplyProducingMessageHandlerWrapper.handleRequestMessage(ReplyProducingMessageHandlerWrapper.java:58)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:134)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:62)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:570)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:520)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.handleMessage(SourcePollingChannelAdapter.java:196)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.messageReceived(AbstractPollingEndpoint.java:444)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:428)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:376)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$null$3(AbstractPollingEndpoint.java:323)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.lambda$execute$0(ErrorHandlingTaskExecutor.java:57)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:55)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$4(AbstractPollingEndpoint.java:320)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Folder is not Open
at com.sun.mail.pop3.POP3Folder.checkOpen(POP3Folder.java:562)
at com.sun.mail.pop3.POP3Folder.getProtocol(POP3Folder.java:592)
at com.sun.mail.pop3.POP3Message.getRawStream(POP3Message.java:154)
at com.sun.mail.pop3.POP3Message.getContentStream(POP3Message.java:251)
at javax.mail.internet.MimePartDataSource.getInputStream(MimePartDataSource.java:78)
at javax.mail.internet.MimeMultipart.parse(MimeMultipart.java:570)
... 35 more
Obviously, the root cause is clear - the POP3 folder is closed. I have seen solutions that would likely be able to handle when just the Java mail classes are used, but none with Spring Integration. My question is how does one properly control when a folder is open or closed using Spring Integration Mail? I realize the Pop3MailReceiver class has a .setAutoCloseFolder() method. Based on the Spring Docs, I assume I need to set that, along something like the following to my handler:
Closeable closeableResource = StaticMessageHeaderAccessor.getCloseableResource(message);
if (closeableResource != null) {
closeableResource.close();
}
However, if I set autoCloseFolder to false, it does not appear as if the message even ever "hits" my handler, so unfortunately being able to close the resource does not even matter at this point. That is, when autoClose is set to false, the 'handleMessageInternal()' method in my handler class is never reached even though there are indeed message on the POP3 server. Instead I just get a bunch of logs like this:
2020-06-26 15:26:54.523 INFO 15348 --- [ scheduling-1] o.s.integration.mail.Pop3MailReceiver : attempting to receive mail from folder [INBOX]
What am I missing?
Thanks.
When ip contains a json-type,
Among the current connection to factory.getConnectionIds() to find the corresponding IP.
Then set the header to send the logic during development.
Through factory.getConnectionIds() found the IP list that is currently connected, I set up a header. but unable to find outbound socket error is occured.
What is the cause?
integration config is...
#Bean
public TcpReceivingChannelAdapter sslAdapter() {
TcpReceivingChannelAdapter adapter = new TcpReceivingChannelAdapter();
adapter.setConnectionFactory(sslServerFactory());
adapter.setOutputChannel(inputWithSSL());
return adapter;
}
#Bean
public TcpSendingMessageHandler sslHandler() {
TcpSendingMessageHandler handler = new TcpSendingMessageHandler();
handler.setConnectionFactory(sslServerFactory());
return handler;
}
#Bean
public AbstractConnectionFactory sslServerFactory() {
int port = Integer.parseInt(inboundPort);
TcpNioServerConnectionFactory factory = new TcpNioServerConnectionFactory(port);
factory.setBacklog(BACKLOG);
factory.setTaskExecutor(taskSchedulerWithSSL());
factory.setLookupHost(false);
factory.setSerializer(echoSerializer);
factory.setDeserializer(echoSerializer);
factory.setTcpNioConnectionSupport(tcpNioSSLConnectionSupport());
// Nagle's algorithm disabled
factory.setSoTcpNoDelay(true);
return factory;
}
#Bean
public IntegrationFlow flowForReceiveSslMessage() {
return IntegrationFlows
.from(sslAdapter)
.<byte[], Boolean>route(
p -> (short) 0 == ByteBuffer.wrap(p, 0, BYTE_LENGTH_OF_SHORT).getShort(),
m -> m.channelMapping(TRUE, INPUT_WITH_SSL_JSON)
.channelMapping(FALSE, INPUT_WITH_SSL_ECHO)).get();
}
#Bean
public IntegrationFlow flowForExtractingSslJson() {
return IntegrationFlows
.from(inputWithSslJson())
.handle(INBOUND_SERVICE, EXTRACT_PAYLOAD_AS_JSON)
.<Map<String, Object>, String>route(
p -> (String) p.get(REQUEST),
m -> m.channelMapping(LOGIN, INPUT_WITH_SSL_LOGIN)
.channelMapping(LOGOUT, INPUT_WITH_SSL_LOGOUT)
.channelMapping(POLICY, INPUT_WITH_SSL_POLICY)
.channelMapping(PUSH_TARGET, INPUT_WITH_SSL_PUSH_TARGET).get();
}
#Bean
public IntegrationFlow flowForHandlingSslNotifyPolicyUpdate() {
return IntegrationFlows.from(inputWithSslPushTarget()).handle(POLICY_SERVICE, RESPONSE_POLICY_UPDATE)
.split(POLICY_SERVICE, SPLIT_MESSAGES)
.channel(outputWithSslJsonBytesToClient()).get();
}
#Bean
public IntegrationFlow flowForConvertingSslJsonToBytesAndSendClient() {
return IntegrationFlows.from(outputWithSslJsonBytesToClient())
.transform(new ObjectToJsonTransformer())
.handle(INBOUND_SERVICE, ATTACH_HEADER_BY_STRING).handle(sslHandler).get();
}
#Bean
public MessageChannel outputWithSsl() {
return MessageChannels.queue(POOL_SIZE).get();
}
#Bean
public MessageChannel inputWithSslJson() {
return MessageChannels.queue(POOL_SIZE).get();
}
#Bean
public MessageChannel inputWithSslPushTarget() {
return MessageChannels.queue(POOL_SIZE).get();
}
#Bean
public MessageChannel outputWithSslJsonBytesToClient() {
return MessageChannels.queue(POOL_SIZE).get();
}
RESPONSE_POLICY_UPDATE and SPLIT_MESSAGES is...
#Override
public Object responsePolicyUpdate(Object payload) throws Exception {
log.debug("notify policy update debug : {}", payload);
Map<String, Object> params = initParam(payload);
Map<String, Object> result = initResult(params);
result.put(RESPONSE, PUSH_TARGET);
result.put(RESULT, SUCCESS);
result.put(REASON, 0);
return result;
}
#Splitter
#Override
#SuppressWarnings("unchecked")
public List<Message<String>> splitMessages(Object payload) throws Exception {
log.debug("split messages debug : {}", payload);
Map<String, Object> params = initParam(payload);
List<String> pushTargetList = (List<String>) params.get(PUSH_TARGET_LIST); // pushTargetList is ip list.
List<Message<String>> messageList = new ArrayList<Message<String>>();
String[] conArray = new String[4];
List<String> sslConnectionIds = sslServerFactory.getOpenConnectionIds();
int sslPort = sslServerFactory.getPort();
for (String con : sslConnectionIds) {
log.debug("## con ip : {}", con);
conArray = con.split(":");
for (String pushTargetIP : pushTargetList) {
if (conArray[0].equals(pushTargetIP)) {
Message<String> message = MessageBuilder.withPayload(params.toString())
.setHeader("ip_connectionId", con).build();
messageList.add(message);
break;
}
}
}
return messageList;
}
debug log is...
The first line is the current connection list.
2016-07-05 14:30:14.664 DEBUG 56092 --- [sk-scheduler-10] c.m.j.policy.service.PolicyServiceImpl : ## con ip : 192.168.3.57:62370:5443:cdeb011d-91f5-46c4-abc9-b68ba13624b3
2016-07-05 14:30:14.672 DEBUG 56092 --- [ask-scheduler-1] o.s.i.ip.tcp.TcpSendingMessageHandler : plainHandler received message: GenericMessage [payload=byte[246], headers={sequenceNumber=1, json__TypeId__=class java.lang.String, sequenceSize=1, ip_connectionId=192.168.3.57:62370:5443:cdeb011d-91f5-46c4-abc9-b68ba13624b3, correlationId=fae71250-bf47-3f64-6ad3-1ce22ef69464, id=c6c097f0-9efb-f0a5-4240-924e06879b7f, contentType=application/json, timestamp=1467696614672}]
2016-07-05 14:30:14.672 ERROR 56092 --- [ask-scheduler-1] o.s.i.ip.tcp.TcpSendingMessageHandler : Unable to find outbound socket for GenericMessage [payload=byte[246], headers={sequenceNumber=1, json__TypeId__=class java.lang.String, sequenceSize=1, ip_connectionId=192.168.3.57:62370:5443:cdeb011d-91f5-46c4-abc9-b68ba13624b3, correlationId=fae71250-bf47-3f64-6ad3-1ce22ef69464, id=c6c097f0-9efb-f0a5-4240-924e06879b7f, contentType=application/json, timestamp=1467696614672}]
2016-07-05 14:30:14.673 DEBUG 56092 --- [ask-scheduler-1] o.s.i.channel.PublishSubscribeChannel : preSend on channel 'errorChannel', message: ErrorMessage [payload=org.springframework.messaging.MessageHandlingException: Unable to find outbound socket, headers={id=273f4477-52cf-645b-d157-e22dc7cc781a, timestamp=1467696614673}]
2016-07-05 14:30:14.673 DEBUG 56092 --- [ask-scheduler-1] o.s.integration.handler.LoggingHandler : (inner bean)#6dc2279c received message: ErrorMessage [payload=org.springframework.messaging.MessageHandlingException: Unable to find outbound socket, headers={id=273f4477-52cf-645b-d157-e22dc7cc781a, timestamp=1467696614673}]
2016-07-05 14:30:14.675 ERROR 56092 --- [ask-scheduler-1] o.s.integration.handler.LoggingHandler : org.springframework.messaging.MessageHandlingException: Unable to find outbound socket
at org.springframework.integration.ip.tcp.TcpSendingMessageHandler.handleMessageInternal(TcpSendingMessageHandler.java:113)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Thanks Gary.
As you say, an error occurs in the method of handleMessageInternal TcpSendingMessageHandler class.
Get connectionId is...
2016-07-06 10:04:28.704 DEBUG 30144 --- [ask-scheduler-4] c.m.j.policy.service.PolicyServiceImpl : ## con ip : 192.168.3.57:53759:5443:bf93680b-13fe-401b-a1eb-5545917f404a
connectionId is not null. But, the result of connections.get (connectionId) is null.
This should not be understood.
This is the TcpSendingMessageHandler class...
/**
* Writes the message payload to the underlying socket, using the specified
* message format.
* #see org.springframework.messaging.MessageHandler#handleMessage(org.springframework.messaging.Message)
*/
#Override
public void handleMessageInternal(final Message<?> message) throws
MessageHandlingException {
if (this.serverConnectionFactory != null) {
// We don't own the connection, we are asynchronously replying
Object connectionId = message.getHeaders().get(IpHeaders.CONNECTION_ID);
TcpConnection connection = null;
if (connectionId != null) {
connection = connections.get(connectionId);
}
if (connection != null) {
try {
connection.send(message);
}
catch (Exception e) {
logger.error("Error sending message", e);
connection.close();
if (e instanceof MessageHandlingException) {
throw (MessageHandlingException) e;
}
else {
throw new MessageHandlingException(message, "Error sending message", e);
}
}
}
else {
logger.error("Unable to find outbound socket for " + message);
throw new MessageHandlingException(message, "Unable to find outbound socket");
}
return;
}
else {
// we own the connection
try {
doWrite(message);
}
catch (MessageHandlingException e) {
// retry - socket may have closed
if (e.getCause() instanceof IOException) {
if (logger.isDebugEnabled()) {
logger.debug("Fail on first write attempt", e);
}
doWrite(message);
}
else {
throw e;
}
}
}
}
This is Message<String> list...
The value obtained by factory.getOpenConnectionIds() method to get into the ip_connectionId.
Why not find a outboud socket?
GenericMessage [payload={result=success, reason=0, response=pushTarget}, headers={ip_connectionId=192.168.3.57:58187:5443:37702eaf-0bbc-44a1-8763-65e841a2f480, id=a1b80cc4-3f56-1b80-9c59-57be98b1031e, timestamp=1467783978378}]
GenericMessage [payload={result=success, reason=0, response=pushTarget}, headers={ip_connectionId=192.168.3.40:53161:5443:693c394c-d3dd-42a3-95ce-692a39a8b603, id=bb49ea99-5e3b-eccf-df3b-7ce03b4bbf73, timestamp=1467783978378}]
I want to POST Logs to "Custom Logs" of Stackdriver. These feature is beta, and maybe therefore it has no description, how to use Logging with Java API on App Engine. Anyway I want to describe my problem: I use this API version:
"com.google.apis:google-api-services-logging:v2beta1-rev10-1.21.0"
So, first I build the Logging Object like this (I hope this is right):
public static Logging createAuthorizedClient() throws IOException {
// Create the credential
HttpTransport transport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleCredential credential = GoogleCredential.getApplicationDefault(transport, jsonFactory);
if (credential.createScopedRequired()) {
credential = credential.createScoped(LoggingScopes.all());
}
return new Logging.Builder(transport, jsonFactory, credential).setApplicationName(SharedConstants.APPLICATION_ID).build();
}
After I get the Logging client, I try to push an Entry to the Log:
LogEntry lEntry = new LogEntry();
lEntry.setTextPayload("I want to see this log!");
WriteLogEntriesRequest writeLogEntriesRequest = new WriteLogEntriesRequest();
writeLogEntriesRequest.setLogName("My Super log");
List<LogEntry> listEntries = new ArrayList<>();
listEntries.add(lEntry);
writeLogEntriesRequest.setEntries(listEntries);
Logging logging = LoggingManager.createAuthorizedClient();
Write write = logging.entries().write(writeLogEntriesRequest);
WriteLogEntriesResponse writeLogResponse = write.execute();
But what I get is:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 OK
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Invalid resource id",
"reason" : "badRequest"
} ],
"message" : "Invalid resource id",
"status" : "INVALID_ARGUMENT"
}
=== UPDATE: WORKING SOLUTION ===
Thanks to mshamma. Here is complete code, how to send the data to the logging:
public boolean send() {
WriteLogEntriesResponse response = null;
try {
final String now = getNowUtc();
final String insertId = "entry-at-" + now;
final Map<String, String> labels = ImmutableMap.of("project_id", SharedConstants.APPLICATION_ID, "name",
"projects/" + SharedConstants.APPLICATION_ID + "/logs/" + this.logName);
Logging service = createAuthorizedClient();
MonitoredResource ressource = new MonitoredResource();
ressource.setType("logging_log");
ressource.setLabels(labels);
LogEntry entry = new LogEntry().setInsertId(insertId).setResource(ressource).setTimestamp(now)
.setJsonPayload(this.entriesMap)
.setLogName("projects/" + SharedConstants.APPLICATION_ID + "/logs/" + this.logName)
.setSeverity(this.severity);
WriteLogEntriesRequest content = (new WriteLogEntriesRequest())
.setEntries(Collections.singletonList(entry));
response = service.entries().write(content).execute();
} catch (Exception e) {
}
return response != null;
}
private static String getNowUtc() {
SimpleDateFormat dateFormatUtc = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'");
dateFormatUtc.setTimeZone(TimeZone.getTimeZone("UTC"));
return dateFormatUtc.format(new Date());
}
This code works fine with the last version of logging api
Thereby the EntriesMap is:
private Map<String, Object> entriesMap;
I ran into the same issue in the unmanaged Python environment. I got things working and I can see at least two issues in your code.
The log name needs to follow the pattern: "projects/<project-id>/logs/<log-id>". See the documentation of the field here: https://cloud.google.com/logging/docs/api/ref_v2beta1/rest/v2beta1/LogEntry#SCHEMA_REPRESENTATION
You should add a resource descriptor both to the log entry (lEntry) and the write log entry request (writeLogEntriesRequest). In the case of GAE, the resource type field should be set to "gae_app" and you must add three labels to the resource that identify your GAE deployment: "project_id", "module_id" and "version_id".
I hope that will help resolve your issue!