The following initializes as socket with spring-integration on a dedicated port.
Goal is to give the application 2 minutes of time to run on that port, or fail is not successful. But the application start should not fail directly of that port is not free on startup!
#Configuration
#EnableIntegration
public class SocketConfiguration {
#Value("${app.port}")
private String port;
#Bean
public TcpConnectionFactoryFactoryBean tcpFactory() {
CompletableFuture.runAsync(() -> SocketUtils.findAvailableTcpPort(port, port)).get(2, TimeUnit.MINUTES);
}
}
Problem: SocketUtils is #Deprecated since Spring Framework 6.
What is a proper replacement for it?
The docs only state as an alternative:
Instead of using SocketUtils to find an available local port for a server, it is recommended that you rely on a server's ability to start on a random port
Well, that's not what I want, neither can I use if, because I'm historically bound to that specific port!
There is a org.springframework.test.util.TestSocketUtils instead. But as Gary said it is better to rely on operation system choice via 0 for port option. Although why would one select a random port in a production code...
To get a real selected port you can listen for a TcpConnectionServerListeningEvent which will bring for you that actual value.
Related
The problem I am trying to solve is figuring out all the external dependencies of an "Etl" java jar. The jar can call out to any number of external web services but there is no real insight to what they are calling out to.
We launch these jars from another java application(e.g. "java -jar jarname.jar"), so that's why I think we might be able to do something to intercept all http requests from the launch jar using a java agent or something like that.
In all, is it possible to intercept all outgoing tcp requests when launching a java jar? I say tcp because we are looking for all outgoing connections database, rabbitmq, http.
The usual strategy is to use tools like wireshark which can be configured to tell you all the details (including the data itself), in a nice user interface, and you can trivially filter on 'only all traffic caused by this specific process'.
You CAN also do this with an agent, but you're signing yourself up for a ton of code work for an end result that is going to be vastly inferior to what tools like wireshark can tell you.
If you want to go the agent route, find every entrypoint call that could possibly set up a TCP connection. Hopefully, that's just Socket, but in practice I bet there is more, and I don't think there's a list out there. Use ASM or bytebuddy or some other bytecode rewrite tool to 'load' every class being defined 'in transit', inspect it, and rewrite any calls to e.g. the Socket constructor to your own utility method that logs this, or wraps this so you can keep track of it. This will be quite a sizable project.
You don't really need an agent for this, though; if you want to know when connections happen, just, in your main, set up a SecurityManager - allow everything, but do some logging in the relevant methods. For example:
class MyNetworkInspectingManager extends SecurityManager {
#Override public void checkAccept(String host, int port) {
log.warn("accepting connection on " + host + ":" + port);
}
#Override public void checkConnect(String host, int port, Object context) {
log.warn("opening connection to " + host + ":" + port);
}
#Override public void checkConnect(String host, int port) {
log.warn("opening connection to " + host + ":" + port);
}
}
and possibly, also override checkListen, checkMulticast.
then, in your main:
public static void main(String[] args) throws Exception {
System.setSecurityManager(new MyNetworkInspectingManager());
}
Still far less useful than wireshark, but it's an hour or so to set this up, vs. the weeks+ to make the agent solution work out.
In my current setup, I'm using the default multicast option of the Hazelcast cluster manager. When I link the instances of my containerized Vertx modules (via Docker networking links), I can see that they are successfully creating Hazelcast cluster. However, when I try publishing events on the event bus from one module, the other module doesn't react to it. I'm not sure how the network settings in the Hazelcast cluster related to the network settings for the event bus.
At the moment, I have the following programmatic configuration for each of my Vert.x module, each deployed inside a docker container.
ClusterManager clusterManager = new HazelcastClusterManager();
VertxOptions vertxOptions = new VertxOptions()
.setClustered(true)
.setClusterManager(clusterManager);
vertxOptions.setEventBusOptions(new EventBusOptions()
.setClustered(true)
.setClusterPublicHost("application"));
The Vert.x Core manual states that I may have to configure clusterPublicHost, and clusterPublicPort for the event bus, but I'm not sure how those relate to the general network topology.
One answer is here https://groups.google.com/d/msg/vertx/_2MzDDowMBM/nFoI_k6GAgAJ
I see this question come up a lot, and what a lot of people miss in
the documentation (myself included) is that Event Bus does not use the
cluster manager to send event bus messages. I.e. in your example with
Hazelcast as the cluster manager, you have the Hazelcast cluster up
and communicating properly (so your Cluster Manager is fine); however,
the Event bus is failing to communicate with your other docker
instances due to one or more of the following:
It is attempting to use an incorrect IP address to the other node (i.e. the IP of the private interface on the Docker instance, not the
publicly mapped one)
It is attempting to communicate on a port Docker is not configured to forward (the event bus picks a dynamic port if you don't specify
one)
What you need to do is:
Tell Vertx the IP address that the other nodes should use to talk to each instance ( using the -cluster-host [command line] ,
setClusterPublicHost [VertXOptions] or "vertx.cluster.public.host"
[System Property] options)
Tell Vertx explicitly the Port to use for event bus communication and ensure Docker is forwarding traffic for those ports ( using the
"vertx.cluster.public.port" [System Property], setClusterPublicPort
[VertXOptions] or -cluster-port [command line] options). In the past,
I have used 15701 because it is easy to remember (just a '1' in fromt
of the Hazelcast ports).
The Event bus only uses the Cluster Manager to manage the IP/Port
information of the other Vertx Instances and the registration of the
Consumers/Producers. The communications are done independently of the
cluster manager, which is why you can have the cluster manager
configured properly and communicating, but still have no Event bus
communications.
You may not need to do both the steps above if both your containers
are running on the same host, but you definitely will once you start
running them on separate hosts.
Something what also can happen, is that vert.x uses the loopback interface, when not specifying the IP which vert.x (not hazelcast) should take to communicate over eventbus. The problem here is, that you don't know which interface is taken to communicate over (loopback, interface with IP, you could even have multiple interfaces with IP).
To overcome this problem, I wrote a method once https://github.com/swisspush/vertx-cluster-watchdog/blob/master/src/main/java/org/swisspush/vertx/cluster/ClusterWatchdogRunner.java#L101
The cluster manager works fine, the cluster manager configuration has to be the same on each node (machine/docker container) in your cluster or don't make any configurations at all (use the default configuration of your cluster manager).
You have to make the event bus configuration be consistent on each node, you have to set the cluster host on each node to be the IP address of this node itself and any arbitrary port number (unless you try to run more than Vert.x instance on the same node you have to choose a different port number for each Vert.x instance).
For example if a node's IP address is 192.168.1.12 then you would do the following:
VertxOptions options = new VertxOptions()
.setClustered(true)
.setClusterHost("192.168.1.12") // node ip
.setClusterPort(17001) // any arbitrary port but make sure no other Vert.x instances using same port on the same node
.setClusterManager(clusterManager);
on another node whose IP address is 192.168.1.56 then you would do the following:
VertxOptions options = new VertxOptions()
.setClustered(true)
.setClusterHost("192.168.1.56") // other node ip
.setClusterPort(17001) // it is ok because this is a different node
.setClusterManager(clusterManager);
found this solution that worked perfectly for me, below is my code snippet (important part is the options.setClusterHost()
public class Runner {
public static void run(Class clazz) {
VertxOptions options = new VertxOptions();
try {
// for docker binding
String local = InetAddress.getLocalHost().getHostAddress();
options.setClusterHost(local);
} catch (UnknownHostException e) { }
options.setClustered(true);
Vertx.clusteredVertx(options, res -> {
if (res.succeeded()) {
res.result().deployVerticle(clazz.getName());
} else {
res.cause().printStackTrace();
}
});
}
}
public class Publisher extends AbstractVerticle {
public static void main(String[] args) {
Runner.run(Publisher.class);
}
...
}
no need to define anything else...
I need to build a (standalone Java) restlet-based service that only listens on localhost, i.e. no requests from network are allowed.
I was trying to do the obvious:
Server srv = new Server(Protocol.HTTPS, "localhost", httpsPort);
component.getServers().add(srv);
But the service still listens on 0.0.0.0. :-(
I went into the code and found that HttpsServerHelper ignores the hostname when creating the service:
this.server = HttpsServer.create(new InetSocketAddress(getHelped().getPort()), 0);
Similar code exists in plain HTTP's HttpServerHelper, where it is even more clear.
My question then is this:
How can I configure Restlet component/service to only listen on localhost?
I don't know which server you use under the hood within your standalone Restlet application. You should use a server connector other than the default one and I recommend you to use the Jetty one.
To do that, simply put the jar of the extension org.restlet.ext.jetty in your classpath.
In this case, using the following code should correspond to your needs:
component.getServers().add(Protocol.HTTP, "localhost", 8182);
Here is the corresponding trace at application startup:
2015-09-03 09:47:22.180:INFO::jetty-7.1.6.v20100715
2015-09-03 09:47:22.211:INFO::Started SelectChannelConnector#localhost:8182
In addition, here is the link in the Restlet documentation regarding Restlet connectors: http://restlet.com/technical-resources/restlet-framework/guide/2.3/core/base/connectors.
Hope it helps you,
Thierry
The easier way to achieve that is to use virtual hosts.
Virtual hosts are the first routing barrier when handling a request, especially it helps routing on a domain.
Here is a sample code that illustrates this:
Component c = new Component();
c.getServers().add(Protocol.HTTP, 8182);
VirtualHost host = new VirtualHost();
host.setHostDomain("localhost");
c.getHosts().add(host);
host.attach(new Restlet() {
#Override
public void handle(Request request, Response response) {
response.setEntity("hello, world", MediaType.TEXT_PLAIN);
}
});
c.start();
Usually, applications are attached on the default host of a component. This default host does nothing, except routing requests based on the context path of the attached application:
c.getDefaultHost().attach("/contextPath1", new Test1Application());
c.getDefaultHost().attach("/contextPath2", new Test2Application());
When you would like to filter calls based on other data than the request's path, virtual host may be the solution.
Here is a diagram that may help you:
http://restlet.com/technical-resources/restlet-framework/tutorials/2.3#part05
I have a webservice which is accepting a POST method with XML. It is working fine then at some random occasion, it fails to communicate to the server throwing IOException with message The target server failed to respond. The subsequent calls work fine.
It happens mostly, when i make some calls and then leave my application idle for like 10-15 min. the first call which I make after that returns this error.
I tried couple of things ...
I setup the retry handler like
HttpRequestRetryHandler retryHandler = new HttpRequestRetryHandler() {
public boolean retryRequest(IOException e, int retryCount, HttpContext httpCtx) {
if (retryCount >= 3){
Logger.warn(CALLER, "Maximum tries reached, exception would be thrown to outer block");
return false;
}
if (e instanceof org.apache.http.NoHttpResponseException){
Logger.warn(CALLER, "No response from server on "+retryCount+" call");
return true;
}
return false;
}
};
httpPost.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, retryHandler);
but this retry never got called. (yes I am using right instanceof clause). While debugging this class never being called.
I even tried setting up HttpProtocolParams.setUseExpectContinue(httpClient.getParams(), false); but no use. Can someone suggest what I can do now?
IMPORTANT
Besides figuring out why I am getting the exception, one of the important concerns I have is why isn't the retryhandler working here?
Most likely persistent connections that are kept alive by the connection manager become stale. That is, the target server shuts down the connection on its end without HttpClient being able to react to that event, while the connection is being idle, thus rendering the connection half-closed or 'stale'. Usually this is not a problem. HttpClient employs several techniques to verify connection validity upon its lease from the pool. Even if the stale connection check is disabled and a stale connection is used to transmit a request message the request execution usually fails in the write operation with SocketException and gets automatically retried. However under some circumstances the write operation can terminate without an exception and the subsequent read operation returns -1 (end of stream). In this case HttpClient has no other choice but to assume the request succeeded but the server failed to respond most likely due to an unexpected error on the server side.
The simplest way to remedy the situation is to evict expired connections and connections that have been idle longer than, say, 1 minute from the pool after a period of inactivity. For details please see the 2.5. Connection eviction policy of the HttpClient 4.5 tutorial.
Accepted answer is right but lacks solution. To avoid this error, you can add setHttpRequestRetryHandler (or setRetryHandler for apache components 4.4) for your HTTP client like in this answer.
HttpClient 4.4 suffered from a bug in this area relating to validating possibly stale connections before returning to the requestor. It didn't validate whether a connection was stale, and this then results in an immediate NoHttpResponseException.
This issue was resolved in HttpClient 4.4.1. See this JIRA and the release notes
Solution: change the ReuseStrategy to never
Since this problem is very complex and there are so many different factors which can fail I was happy to find this solution in another post: How to solve org.apache.http.NoHttpResponseException
Never reuse connections:
configure in org.apache.http.impl.client.AbstractHttpClient:
httpClient.setReuseStrategy(new NoConnectionReuseStrategy());
The same can be configured on a org.apache.http.impl.client.HttpClientBuilder builder:
builder.setConnectionReuseStrategy(new NoConnectionReuseStrategy());
Although accepted answer is right, but IMHO is just a workaround.
To be clear: it's a perfectly normal situation that a persistent connection may become stale. But unfortunately it's very bad when the HTTP client library cannot handle it properly.
Since this faulty behavior in Apache HttpClient was not fixed for many years, I definitely would prefer to switch to a library that can easily recover from a stale connection problem, e.g. OkHttp.
Why?
OkHttp pools http connections by default.
It gracefully recovers from situations when http connection becomes stale and request cannot be retried due to being not idempotent (e.g. POST). I cannot say it about Apache HttpClient (mentioned NoHttpResponseException).
Supports HTTP/2.0 from early drafts and beta versions.
When I switched to OkHttp, my problems with NoHttpResponseException disappeared forever.
Nowadays, most HTTP connections are considered persistent unless declared otherwise. However, to save server ressources the connection is rarely kept open forever, the default connection timeout for many servers is rather short, for example 5 seconds for the Apache httpd 2.2 and above.
The org.apache.http.NoHttpResponseException error comes most likely from one persistent connection that was closed by the server.
It's possible to set the maximum time to keep unused connections open in the Apache Http client pool, in milliseconds.
With Spring Boot, one way to achieve this:
public class RestTemplateCustomizers {
static public class MaxConnectionTimeCustomizer implements RestTemplateCustomizer {
#Override
public void customize(RestTemplate restTemplate) {
HttpClient httpClient = HttpClientBuilder
.create()
.setConnectionTimeToLive(1000, TimeUnit.MILLISECONDS)
.build();
restTemplate.setRequestFactory(
new HttpComponentsClientHttpRequestFactory(httpClient));
}
}
}
// In your service that uses a RestTemplate
public MyRestService(RestTemplateBuilder builder ) {
restTemplate = builder
.customizers(new RestTemplateCustomizers.MaxConnectionTimeCustomizer())
.build();
}
This can happen if disableContentCompression() is set on a pooling manager assigned to your HttpClient, and the target server is trying to use gzip compression.
Same problem for me on apache http client 4.5.5
adding default header
Connection: close
resolve the problem
Use PoolingHttpClientConnectionManager instead of BasicHttpClientConnectionManager
BasicHttpClientConnectionManager will make an effort to reuse the connection for subsequent requests with the same route. It will, however, close the existing connection and re-open it for the given route.
I have faced same issue, I resolved by adding "connection: close" as extention,
Step 1: create a new class ConnectionCloseExtension
import com.github.tomakehurst.wiremock.common.FileSource;
import com.github.tomakehurst.wiremock.extension.Parameters;
import com.github.tomakehurst.wiremock.extension.ResponseTransformer;
import com.github.tomakehurst.wiremock.http.HttpHeader;
import com.github.tomakehurst.wiremock.http.HttpHeaders;
import com.github.tomakehurst.wiremock.http.Request;
import com.github.tomakehurst.wiremock.http.Response;
public class ConnectionCloseExtension extends ResponseTransformer {
#Override
public Response transform(Request request, Response response, FileSource files, Parameters parameters) {
return Response.Builder
.like(response)
.headers(HttpHeaders.copyOf(response.getHeaders())
.plus(new HttpHeader("Connection", "Close")))
.build();
}
#Override
public String getName() {
return "ConnectionCloseExtension";
}
}
Step 2: set extension class in wireMockServer like below,
final WireMockServer wireMockServer = new WireMockServer(options()
.extensions(ConnectionCloseExtension.class)
.port(httpPort));
I have a Netty TCP server, and I want to reject/refuse incoming connection attempts selectively (based on their remote address). I guess I have to use ServerBootstrap.setParentHandler(ChannelHandler), but what do I do in the ChannelHandler? What event am I handling? How do I refuse the connection?
As Norman said, there is no way to refuse the connection, but you can close it immediately by adding a Netty's IpFilterHandler to server pipeline as the first handler. It will also stop propagating the upstream channel state events for filtered connection too.
#ChannelHandler.Sharable
public class MyFilterHandler extends IpFilteringHandlerImpl {
private final Set<InetSocketAddress> deniedRemoteAddress;
public MyFilterHandler(Set<InetSocketAddress> deniedRemoteAddress) {
this.deniedRemoteAddress = deniedRemoteAddress;
}
#Override
protected boolean accept(ChannelHandlerContext ctx, ChannelEvent e, InetSocketAddress inetSocketAddress) throws Exception {
return !deniedRemoteAddress.contains(inetSocketAddress);
}
}
if you have list of patterns of IP address to block, you can use IpFilterRuleHandler,
//Example: allow only localhost:
new IPFilterRuleHandler().addAll(new IpFilterRuleList("+n:localhost, -n:*"))
If you have several network interfaces and you want to accept connections from one interface only you just need to set the local address in ServerBootstrap. This may be enough if your server is running in a machine that's connected to several networks and you want to serve only one of them. In this case any connection attempts from the other networks would be refused by the OS.
Once you have a connection in the application layer it's too late to refuse it. The best you can do is close it immediately.
This is enough if for example you want the server available only on localhost and invisible to the outside world: the loopback network 127.0.0.0/8 is served by a separate interface.
After having looked at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink in the Netty sources, I am fairly certain that Netty accepts all incoming connections, and there is no way to refuse them (but, of course, they can be closed after being accepted).