Gary Russell kindly answered a previous question of mine about Spring Integration udp flows. Moving from there, I have stumbled upon an issue with ports.
The Spring Integration documentation says that you can put 0 to the inbound channel adapter port, and the OS will select an available port for the adapter, which can be retrieved at runtime invoking getPort() on the adapter object. The problem is that at runtime I just get a 0 if I try to retrieve the port programmatically.
Here's "my" code (i.e. a slightly modified version of Russel's answer to my previous question for Spring Integration 4.3.12, which I am currently using).
#SpringBootApplication
public class TestApp {
private final Map<Integer, IntegrationFlowRegistration> registrations = new HashMap<>();
#Autowired
private IntegrationFlowContext flowContext;
public static void main(String[] args) {
SpringApplication.run(TestApp.class, args);
}
#Bean
public PublishSubscribeChannel channel() {
return new PublishSubscribeChannel();
}
#Bean
public TestData test() {
return new TestData();
}
#Bean
public ApplicationRunner runner() {
return args -> {
UnicastReceivingChannelAdapter source;
source = makeANewUdpInbound(0);
makeANewUdpOutbound(source.getPort());
Thread.sleep(5_000);
channel().send(MessageBuilder.withPayload("foo\n").build());
this.registrations.values().forEach(r -> {
r.stop();
r.destroy();
});
this.registrations.clear();
makeANewUdpInbound(1235);
makeANewUdpOutbound(1235);
Thread.sleep(5_000);
channel().send(MessageBuilder.withPayload("bar\n").build());
this.registrations.values().forEach(r -> {
r.stop();
r.destroy();
});
this.registrations.clear();
};
}
public UnicastSendingMessageHandler makeANewUdpOutbound(int port) {
System.out.println("Creating an adapter to send to port " + port);
UnicastSendingMessageHandler adapter = new UnicastSendingMessageHandler("localhost", port);
IntegrationFlow flow = IntegrationFlows.from(channel())
.handle(adapter)
.get();
IntegrationFlowRegistration registration = flowContext.registration(flow).register();
registrations.put(port, registration);
return adapter;
}
public UnicastReceivingChannelAdapter makeANewUdpInbound(int port) {
System.out.println("Creating an adapter to receive from port " + port);
UnicastReceivingChannelAdapter source = new UnicastReceivingChannelAdapter(port);
IntegrationFlow flow = IntegrationFlows.from(source)
.<byte[], String>transform(String::new)
.handle(System.out::println)
.get();
IntegrationFlowRegistration registration = flowContext.registration(flow).register();
registrations.put(port, registration);
return source;
}
}
The output I read is
Creating an adapter to receive from port 0
Creating an adapter to send to port 0
Creating an adapter to receive from port 1235
Creating an adapter to send to port 1235
GenericMessage [payload=bar, headers={ip_packetAddress=127.0.0.1/127.0.0.1:54374, ip_address=127.0.0.1, id=c95d6255-e63a-433d-3723-c389fe66b060, ip_port=54374, ip_hostname=127.0.0.1, timestamp=1517220716983}]
I suspect the library did create adapters on OS-chosen free ports, but I am unable to retrieve the assigned port.
The port is assigned asynchronously; you need to wait until the port is actually assigned. Something like...
int n = 0;
while (n++ < 100 && ! source.isListening()) {
Thread.sleep(100;
}
if (!source.isListening()) {
// failed to start in 10 seconds.
}
We should probably enhance the adapter to emit an event when the port is ready. Feel free to open an 'Improvement' JIRA Issue.
Related
I am trying to build a module to be deployed on multiple nodes using Spring boot. Due to time constraints of the specific application, I have to use UDP and cannot rely on the easier-to-use REST facilities that Spring provides.
I have to be able to send datagrams to a set of nodes that may vary in time (i.e. the set may grow or shrink, or some nodes may move to new ip/port "coordinates"). Communication must be unicast.
I have been reading the official documentation about TCP and UDP support TCP and UDP support, but it is rather... compact, and opaque. The javadocs on the org.springframework.integration classes are also rather brief for that matter.
For what I could understand, an "inbound" channel is used to send a packet, while an outbound channel is used to receive packets.
I haven't been able so far to find an answer to the following issues for inbound (i.e. "send" channels, if I understood well):
- How can I create more channels at runtime, to send packets to multiple destinations?
- If a host gets moved, should I just destroy the channel and set up a new one, or may I change a channel's parameters (destination ip/port) at runtime?
For outbound channels ("receive" channels if I understood well), I have similar questions to the above, as in:
- How do I set up multiple channels at runtime?
- How do I change destination for an existing channel at runtime, not to have to tear it down and set it up anew?
- Should I just open/close "raw" UDP sockets instead?
You have inbound and outbound reversed.
Here's an example that should provide you with what you need; it uses a pub/sub channel to broadcast...
#SpringBootApplication
public class So48213450Application {
private final Map<Integer, IntegrationFlowRegistration> registrations = new HashMap<>();
public static void main(String[] args) {
SpringApplication.run(So48213450Application.class, args);
}
#Bean
public PublishSubscribeChannel channel() {
return new PublishSubscribeChannel();
}
#Bean
public ApplicationRunner runner(PublishSubscribeChannel channel) {
return args -> {
makeANewUdpAdapter(1234);
makeANewUdpAdapter(1235);
channel.send(MessageBuilder.withPayload("foo\n").build());
registrations.values().forEach(r -> {
r.stop();
r.destroy();
});
};
}
#Autowired
private IntegrationFlowContext flowContext;
public void makeANewUdpAdapter(int port) {
System.out.println("Creating an adapter to send to port " + port);
IntegrationFlow flow = IntegrationFlows.from(channel())
.handle(Udp.outboundAdapter("localhost", port))
.get();
IntegrationFlowRegistration registration = flowContext.registration(flow).register();
registrations.put(port, registration);
}
}
result:
$ nc -u -l 1234 &
[1] 56730
$ nc -u -l 1235 &
[2] 56739
$ jobs
[1]- Running nc -u -l 1234 &
[2]+ Running nc -u -l 1235 &
$ foo
foo
You can't change parameters at runtime, you would have to create new ones.
EDIT
In response to your comments below...
You can't mix and match spring integration jars (2.1.x and 5.0.x); they must all be with the same version. My example above used Boot 2.0.0.M7 (boot 2 is scheduled to be released next month).
The Udp factory class was added to spring-integration-ip in 5.0.0.
Here is a similar example (which also adds receiving adapters) for boot 1.5.9 and spring integration 4.3.13...
#SpringBootApplication
public class So482134501Application {
private final Map<Integer, IntegrationFlowRegistration> registrations = new HashMap<>();
#Autowired
private IntegrationFlowContext flowContext;
public static void main(String[] args) {
SpringApplication.run(So482134501Application.class, args);
}
#Bean
public PublishSubscribeChannel channel() {
return new PublishSubscribeChannel();
}
#Bean
public ApplicationRunner runner(PublishSubscribeChannel channel) {
return args -> {
makeANewUdpInbound(1234);
makeANewUdpInbound(1235);
makeANewUdpOutbound(1234);
makeANewUdpOutbound(1235);
Thread.sleep(5_000);
channel.send(MessageBuilder.withPayload("foo\n").build());
this.registrations.values().forEach(r -> {
r.stop();
r.destroy();
});
this.registrations.clear();
};
}
public void makeANewUdpOutbound(int port) {
System.out.println("Creating an adapter to send to port " + port);
IntegrationFlow flow = IntegrationFlows.from(channel())
.handle(new UnicastSendingMessageHandler("localhost", port))
.get();
IntegrationFlowRegistration registration = flowContext.registration(flow).register();
registrations.put(port, registration);
}
public void makeANewUdpInbound(int port) {
System.out.println("Creating an adapter to receive from port " + port);
IntegrationFlow flow = IntegrationFlows.from(new UnicastReceivingChannelAdapter(port))
.<byte[], String>transform(String::new)
.handle(System.out::println)
.get();
IntegrationFlowRegistration registration = flowContext.registration(flow).register();
registrations.put(port, registration);
}
}
result:
GenericMessage [payload=foo
, headers={ip_packetAddress=localhost/127.0.0.1:54881, ip_address=127.0.0.1, id=db7dae61-078c-5eb6-dde4-f83fc6c591d1, ip_port=54881, ip_hostname=localhost, timestamp=1515764556722}]
GenericMessage [payload=foo
, headers={ip_packetAddress=localhost/127.0.0.1:54880, ip_address=127.0.0.1, id=d1f79e79-569b-637b-57c5-549051f1b031, ip_port=54880, ip_hostname=localhost, timestamp=1515764556722}]
I am trying to register a simple REST service on int port,
to ZooKeeper server at localhost:2181.
I checked path ls / using zooClient too.
Any ideas?
private static void registerInZookeeper(int port) throws Exception {
CuratorFramework curatorFramework = CuratorFrameworkFactory
.newClient("localhost:2181", new RetryForever(5));
curatorFramework.start();
ServiceInstance<Object> serviceInstance = ServiceInstance.builder()
.address("localhost")
.port(port)
.name("worker")
.uriSpec(new UriSpec("{scheme}://{address}:{port}"))
.build();
ServiceDiscoveryBuilder.builder(Object.class)
.basePath("myNode")
.client(curatorFramework)
.thisInstance(serviceInstance)
.build()
.start();
Optional.ofNullable(curatorFramework.checkExists().forPath("/zookeeper")).ifPresent(System.out::println);
Optional.ofNullable(curatorFramework.checkExists().forPath("/myNode")).ifPresent(System.out::println);
}
I kept receiving Received packet at server of unknown type 15 from Zoo Server, because of compatibility issues
the registration code here looks correct. In order to print registered instances the following code can be executed:
Optional.ofNullable(curatorFramework.getChildren().forPath("/myNode/worker"))
.orElse(Collections.emptyList())
.forEach(childNode -> {
try {
System.out.println(childNode);
System.out.println(new String(curatorFramework.getData().forPath("/myNode/worker/" + childNode)));
} catch (Exception e) {
e.printStackTrace();
}
});
The result will be like
07:23:12.353 INFO [main-EventThread] ConnectionStateManager:228 - State change: CONNECTED
48202336-e89b-4724-912b-89620f7c9954
{"name":"worker","id":"48202336-e89b-4724-912b-89620f7c9954","address":"localhost","port":1000,"sslPort":null,"payload":null,"registrationTimeUTC":1515561792319,"serviceType":"DYNAMIC","uriSpec":{"parts":[{"value":"scheme","variable":true},{"value":"://","variable":false},{"value":"address","variable":true},{"value":":","variable":false},{"value":"port","variable":true}]}}
Creating your curator framework with zk34 (the version used by kafka) compatibility should fix your problem
private CuratorFramework buildFramework(String ip) {
RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);
return CuratorFrameworkFactory.builder().zk34CompatibilityMode(true).connectString(ip + ":2181")
.retryPolicy(retryPolicy).build();
}
Please note that curator will just try its best and some new methods (eg. creatingParentsIfNeeded (ok) vs creatingParentContainersIfNeeded (ko)) will fail.
I am building an application using Spring Integration which is used to send files from one FTP server (source) to another FTP server (target). I first send files from source to the local directory using the inbound adapter and then send files from the local directory to the target using the outbound adapter.
My code seems to be working fine and I am able to achieve my goal but my problem is when the connection is reset to the target FTP server during the transfer of files, then the transfer of files don't continue after the connection starts working.
I used the Java configurations using inbound and outbound adapters. Can anyone please tell me if it is possible to resume my transfer of files somehow after the connection reset?
P.S: I am a beginner at Spring, so correct me if I have done something wrong here. Thanks
AppConfig.java:
#Configuration
#Component
public class FileTransferServiceConfig {
#Autowired
private ConfigurationService configurationService;
public static final String FILE_POLLING_DURATION = "5000";
#Bean
public SessionFactory<FTPFile> sourceFtpSessionFactory() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost(configurationService.getSourceHostName());
sf.setPort(Integer.parseInt(configurationService.getSourcePort()));
sf.setUsername(configurationService.getSourceUsername());
sf.setPassword(configurationService.getSourcePassword());
return new CachingSessionFactory<FTPFile>(sf);
}
#Bean
public SessionFactory<FTPFile> targetFtpSessionFactory() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost(configurationService.getTargetHostName());
sf.setPort(Integer.parseInt(configurationService.getTargetPort()));
sf.setUsername(configurationService.getTargetUsername());
sf.setPassword(configurationService.getTargetPassword());
return new CachingSessionFactory<FTPFile>(sf);
}
#MessagingGateway
public interface MyGateway {
#Gateway(requestChannel = "toFtpChannel")
void sendToFtp(Message message);
}
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
FtpInboundFileSynchronizer fileSynchronizer = new FtpInboundFileSynchronizer(sourceFtpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory(configurationService.getSourceDirectory());
fileSynchronizer.setFilter(new FtpSimplePatternFileListFilter(
configurationService.getFileMask()));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "ftpChannel",
poller = #Poller(fixedDelay = FILE_POLLING_DURATION ))
public MessageSource<File> ftpMessageSource() {
FtpInboundFileSynchronizingMessageSource source =
new FtpInboundFileSynchronizingMessageSource(ftpInboundFileSynchronizer());
source.setLocalDirectory(new File(configurationService.getLocalDirectory()));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel")
public MessageHandler targetHandler() {
FtpMessageHandler handler = new FtpMessageHandler(targetFtpSessionFactory());
handler.setRemoteDirectoryExpression(new LiteralExpression(
configurationService.getTargetDirectory()));
return handler;
}
}
Application.java:
#SpringBootApplication
public class Application {
public static ConfigurableApplicationContext context;
public static void main(String[] args) {
context = new SpringApplicationBuilder(Application.class)
.web(false)
.run(args);
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel")
public MessageHandler sourceHandler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
Object payload = message.getPayload();
System.out.println("Payload: " + payload);
if (payload instanceof File) {
File file = (File) payload;
System.out.println("Trying to send " + file.getName() + " to target");
}
MyGateway gateway = context.getBean(MyGateway.class);
gateway.sendToFtp(message);
}
};
}
}
First of all it isn't clear what is that sourceHandler for, but you really should be sure that it is subscribed (or targetHandler) to proper channel.
I somehow believe that in your target code the targetHandler is really subscribed to the toFtpChannel.
Anyway that isn't related.
I think the problem here is exactly with the AcceptOnceFileListFilter and error. So, filter work first during directory scan and load all the local files to the in-memory queue for performance reason. Then all of them are sent to the channel for processing. When we reach the targetHandler and got an exception, we just silently got away to the global errorChannel loosing the fact that file hasn't been transferred. And this happens with all the remaining files in memory. I think the transfer is resumed anyway but it is going work already only for new files in the remote directory.
I suggest you to add ExpressionEvaluatingRequestHandlerAdvice to the targetHandler definition (#ServiceActivator(adviceChain)) and in case of error call the AcceptOnceFileListFilter.remove(File):
/**
* Remove the specified file from the filter so it will pass on the next attempt.
* #param f the element to remove.
* #return true if the file was removed as a result of this call.
*/
boolean remove(F f);
This way you remove the failed files from the filter and it will be picked up on the next poll task. You have to make AcceptOnceFileListFilter to be able to get an access to it from the onFailureExpression. The file is the payload of request message.
EDIT
The sample for the ExpressionEvaluatingRequestHandlerAdvice:
#Bean
public Advice expressionAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setOnFailureExpressionString("#acceptOnceFileListFilter.remove(payload)");
advice.setTrapException(true);
return advice;
}
...
#ServiceActivator(inputChannel = "ftpChannel", adviceChain = "expressionAdvice")
Everything rest you can get from their JavaDocs.
I know spring integration has TcpInboundGateway and ByteArrayStxEtxSerializer to handle data coming through TCP port.
ByteArrayStxEtxSerializer works great if the TCP server needs to read all the data sent from the client and then processes it. (request and response model) I am using single-use=false so that multiple requests can be processed in the same connection.
For example if the client sends 0x02AAPL0x03 then Server can send the AAPL price.
My TCP Server is working if the client sends 0x02AAPL0x030x02GOOG0x03. It sends the price of AAPL and GOOG price.
Sometimes clients can send EOT (0x04). If the client sends EOT, I would like to close the socket connection.
For example: Client request can be 0x02AAPL0x030x02GOOG0x03 0x020x040x03. Note EOT came in the last packet.
I know ByteArrayStxEtxSerializer deserializer can be customized to read the bytes sent by the client.
is deserializer good place to close socket connection? if not, how should spring integration framework be notified to close socket connection?
Please help.
Here is my spring configuration:
<int-ip:tcp-connection-factory id="crLfServer"
type="server"
port="${availableServerSocket}"
single-use="false"
so-timeout="10000"
using-nio="false"
serializer="connectionSerializeDeserialize"
deserializer="connectionSerializeDeserialize"
so-linger="2000"/>
<bean id="connectionSerializeDeserialize" class="org.springframework.integration.ip.tcp.serializer.ByteArrayStxEtxSerializer"/>
<int-ip:tcp-inbound-gateway id="gatewayCrLf"
connection-factory="crLfServer"
request-channel="serverBytes2StringChannel"
error-channel="errorChannel"
reply-timeout="10000"/> <!-- reply-timeout works on inbound-gateway -->
<int:channel id="toSA" />
<int:service-activator input-channel="toSA"
ref="myService"
method="prepare"/>
<int:object-to-string-transformer id="serverBytes2String"
input-channel="serverBytes2StringChannel"
output-channel="toSA"/>
<int:transformer id="errorHandler"
input-channel="errorChannel"
expression="payload.failedMessage.payload + ':' + payload.cause.message"/>
UPDATE:
Adding throw new SoftEndOfStreamException("Stream closed") to close the stream in serializer works and I can see the CLOSED log entry in EventListener. When the server closes the connection, I expect to receive java.io.InputStream.read() as -1 in the client. But the client is receiving the
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at sun.nio.cs.StreamDecoder.read0(StreamDecoder.java:107)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:93)
at java.io.InputStreamReader.read(InputStreamReader.java:151)
is there anything else to close the connection on the server side and propagate it to client?
I appreciate your help.
Thank you
The deserializer doesn't have access to the socket, just the input stream; closing it would probably work, but you will likely get a lot of noise in the log.
The best solution is to throw a SoftEndOfStreamException; that signals that the socket should be closed and everything cleaned up.
EDIT
Add a listener to detect/log the close...
#SpringBootApplication
public class So40471456Application {
public static void main(String[] args) throws Exception {
ConfigurableApplicationContext context = SpringApplication.run(So40471456Application.class, args);
Socket socket = SocketFactory.getDefault().createSocket("localhost", 1234);
socket.getOutputStream().write("foo\r\n".getBytes());
socket.close();
Thread.sleep(10000);
context.close();
}
#Bean
public EventListener eventListener() {
return new EventListener();
}
#Bean
public TcpNetServerConnectionFactory server() {
return new TcpNetServerConnectionFactory(1234);
}
#Bean
public TcpReceivingChannelAdapter inbound() {
TcpReceivingChannelAdapter adapter = new TcpReceivingChannelAdapter();
adapter.setConnectionFactory(server());
adapter.setOutputChannelName("foo");
return adapter;
}
#ServiceActivator(inputChannel = "foo")
public void syso(byte[] in) {
System.out.println(new String(in));
}
public static class EventListener implements ApplicationListener<TcpConnectionCloseEvent> {
private final Log logger = LogFactory.getLog(getClass());
#Override
public void onApplicationEvent(TcpConnectionCloseEvent event) {
logger.info(event);
}
}
}
With XML, just add a <bean/> for your listener class.
Result:
foo
2016-11-07 16:52:04.133 INFO 29536 --- [pool-1-thread-2] c.e.So40471456Application$EventListener : TcpConnectionCloseEvent
[source=org.springframework.integration.ip.tcp.connection.TcpNetConnection#118a7548],
[factory=server, connectionId=localhost:50347:1234:b9fcfaa9-e92c-487f-be59-1ed7ebd9312e]
**CLOSED**
EDIT2
It worked as expected for me...
#SpringBootApplication
public class So40471456Application {
public static void main(String[] args) throws Exception {
ConfigurableApplicationContext context = SpringApplication.run(So40471456Application.class, args);
Socket socket = SocketFactory.getDefault().createSocket("localhost", 1234);
socket.getOutputStream().write("foo\r\n".getBytes());
try {
System.out.println("\n\n\n" + socket.getInputStream().read() + "\n\n\n");
context.getBean(EventListener.class).latch.await(10, TimeUnit.SECONDS);
}
finally {
socket.close();
context.close();
}
}
#Bean
public EventListener eventListener() {
return new EventListener();
}
#Bean
public TcpNetServerConnectionFactory server() {
TcpNetServerConnectionFactory server = new TcpNetServerConnectionFactory(1234);
server.setDeserializer(is -> {
throw new SoftEndOfStreamException();
});
return server;
}
#Bean
public TcpReceivingChannelAdapter inbound() {
TcpReceivingChannelAdapter adapter = new TcpReceivingChannelAdapter();
adapter.setConnectionFactory(server());
adapter.setOutputChannelName("foo");
return adapter;
}
public static class EventListener implements ApplicationListener<TcpConnectionCloseEvent> {
private final Log logger = LogFactory.getLog(getClass());
private final CountDownLatch latch = new CountDownLatch(1);
#Override
public void onApplicationEvent(TcpConnectionCloseEvent event) {
logger.info(event);
latch.countDown();
}
}
}
Result:
2016-11-08 08:27:25.964 INFO 86147 --- [ main] com.example2.So40471456Application : Started So40471456Application in 1.195 seconds (JVM running for 1.764)
-1
2016-11-08 08:27:25.972 INFO 86147 --- [pool-1-thread-2] c.e.So40471456Application$EventListener : TcpConnectionCloseEvent [source=org.springframework.integration.ip.tcp.connection.TcpNetConnection#fee3774], [factory=server, connectionId=localhost:54984:1234:f79a6826-0336-4823-8844-67054903a094] **CLOSED**
The only thing I've found in the API is localConsumer so my idea was to register the consumer twice - once using consumer and once using localConsumer with the address prefixed by local:.
Here's an example:
public class VertxLocal {
public static void main(String[] args) {
final UUID id = UUID.randomUUID();
Config config = new Config();
config.getNetworkConfig().setPortAutoIncrement(true);
HazelcastClusterManager cm = new HazelcastClusterManager(config);
// This is io.vertx.rxjava.core.Vertx
Vertx.clusteredVertxObservable(new VertxOptions().setClusterManager(cm))
.subscribe(vertx -> {
EventBus bus = vertx.eventBus();
// Register the "public" consumer
bus.consumer("test")
.toObservable()
.subscribe(message -> {
message.reply(id.toString());
});
// Register the local consumer
bus.localConsumer("local:test")
.toObservable()
.subscribe(message -> {
message.reply(id.toString());
});
// Periodically retrieve replies from the local consumer
vertx.setPeriodic(1, i -> {
bus.sendObservable("local:test", id.toString())
.subscribe(reply -> {
UUID fromId = UUID.fromString((String)reply.body());
if(fromId.equals(id)) {
System.out.println("Received message from local handler");
} else {
System.out.println("Received message from remote handler");
}
});
});
});
}
}
This works, but it's rather ugly to register each consumer twice - I'd rather only choose if should be a local message or not when sending the message.
Is there any way of doing this without registering a local consumer - something like localSend (though I can't see anything like that in the API)?