I'm using proxies with htmlunit, my proxy list contain mixture of both http and socks, I dont know if the next selected proxy to be passed to htmlunit is type http or socks, will htmlunit automatically determine the type and use the appropriate rule for connecting through that proxy?
I've made an application which used mixture of proxies as well, but it was a while ago. In that version of HtmlUnit, it required being explicitly told if the proxy was SOCKS or not, otherwise it assumed it was a HTTP proxy. I looked briefly in the change logs, and found nothing indicating that this had changed, so the answer should be no, it will assume that the proxy is HTTP if you don't tell HtmlUnit that the proxy is SOCKS.
To check what type a proxy is, one can use something like:
SocketAddress addr = new InetSocketAddress("proxyAddress", port);
Proxy proxy = new Proxy(Proxy.Type.HTTP, addr); //or Proxy.Type.SOCKS
URL url = new URL("http://google.com");
URConnection conn = url.openConnection(proxy);
If the code fails (i.e throws an exception), then the proxy is most likely either dead or SOCKS. (HtmlUnit will throw an exception in the first case anyway, or you can perform the same test again with Proxy.Type.SOCKS if you aren't certain the proxy is alive.)
Check out Java Networking and Proxies.
It talks about multiple strategies for setting proxies. It also gives option to provide multipe proxy to same connection through proxy selector.
The other two answers are about how to generically use a proxy in a Java program, but it is a little bit different with HtmlUnit. Setting the proxy on the Java process does nothing; instead you want the simulated browser to use the proxy.
int myProxyPort=8080;
WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER_8, "myproxyhost.com", myProxyPort);
This framework detects the proxy type for you and can instantiate a HtmlUnit WebClient with correct Socks/Http/Https proxy configuration for you: https://github.com/subes/invesdwin-webproxy
Essentially it automates the trial&error approach when maintaining its proxy list for you. So you don't have to worry about that in your own code.
Related
I am using spring boot to make a mock of one of our more complicated services. While all of the standard cases are easily done there is one that is causing me some troubles. It is theoretically possible for the application I am mocking to crash and close the connection without sending a response.
I tried several things to achieve this in spring boot without actually having my mock to crash. This includes throwing exceptions that go into an exception handler and from there do not properly response, however so far that either has generated an error response by spring or somehow resolved to an empty response.
Is there an option to have a method in a #Conroller to cause a closed connection without any response?
Closing the connection is the responsibility of HTTP specification and protocol. You cannot enforce it programmatically. Connection negotitation is happening between HTTP Client and HTTP Server.
Although you can try interrupting current Thread or setting header Connection: close, but you should not be messing around with that part of the processing of a Connection and Request. Your HTTP server can start behaving unexpectedly.
Try a different approach. If you need to simulate a closed connection you can programmatically allocate a new instance of HTTP server, send a request to it, put request processing on hold and kill an instance is a separate thread. I'm sure you'll find a better way for this, just get to the root of the problem from a different angle.
You can create method like this:
#ResponseBody
#RequestMapping("/your-url")
public String test() {
return null;
}
you can also change #Controller to #RestController and remove the #ResponseBody
My application has two outgoing SOAP connections. For those, I want to implement TLS. Both are created using CXF.
The javax.xml.ws.Service.getPort() returns an individual bindingProvider (both connections use their own WSDL) but both use the same org.apache.cxf.bus.spring.SpringBus instance.
Before using the bindingProvider, I set the TLS client parameters on the Conduit:
Client client = ClientProxy.getClient(bindingProvider); // different
HTTPConduit httpConduit = (HTTPConduit) client.getConduit(); // same for both connections
TLSClientParameters tlsClientParameters = new TLSClientParameters();
tlsClientParameters.setTrustManagers(getTrustmanagers());
httpConduit.setTlsClientParameters(tlsClientParameters);
The issue is, the retrieved client is different for both connections, but the conduit is the same object. Thus, when I set the parameters for the second connection on the same object, I overwrite the priorly set settings.
The FAQ answer if CXF is threadsafe with "yes" and a number of exceptions. I think the second exception applies here. It says:
CXF answer: CXF proxies are thread safe for MANY use cases. The exceptions are:
[...]
Settings on the conduit - if you use code or configuration to directly manipulate the conduit (like to set TLS settings or similar), those are not thread safe. The conduit is per-instance and thus those settings would be shared. Also, if you use the FailoverFeature and LoadBalanceFeatures, the conduit is replaced on the fly. Thus, settings set on the conduit could get lost before being used on the setting thread.
[...]
For the conduit issues, you COULD install a new ConduitSelector that uses a thread local or similar. That's a bit complex though.
I'm not entirely sure if thread safety is my issue. I create the two connections each in their own component. Springs uses only one thread for initializing all the components, so both connections are initialized by the same thread. But afterwards, the connection is used threads from a pool. Overwriting the settings happens during intialization, so before sending actual SOAP messages using different threads.
When the Conduit is created in org.apache.cxf.endpoint.AbstractConduitSelector#getSelectedConduit, it is done using the SpringBus which is the same instance for both objects.
So, the FAQ tell me to use my own custom ConduitSelector.
I tried setting it before the initialization above:
Client client = ClientProxy.getClient(bindingProvider);
client.setConduitSelector(
new UpfrontConduitSelector(
new URLConnectionHTTPConduit(client.getBus(),
client.getEndpoint().getEndpointInfo())));
and I tried the same after the initialization. In both cases, after setting the conduit selector, when something uses the BindingProvider (which is a Proxy-object), it gets a NullPointerException although the object is not null.
My issue here is to either get the custom conduit selector running or to see that my issue can be solved completely differently or just to get some inspiration :)
Some guy on SO seems to have solved this here, but the answer to his question is not helping me.
I found a solution.
The issue indeed had nothing to do with multithreading, but with the way the SpringBus is wired into my objects and how the Conduit is created from it.
The solution was to give each service its own SpringBus.
So before I create each SOAP-service by calling its c'tor in javax.xml.ws.Service, I do
BusFactory bf = BusFactory.newInstance();
Bus b = bf.createBus();
BusFactory.setThreadDefaultBus(b);
which sets a new threadlocal default bus which is then used for the service created. Thus, my two services each have their own SpringBus and they both create their own Conduit.
This works because each service is a spring #Component and all spring components are created by the main thread. So there is only one thread and no way this code will not be executed sequentially.
My application uses some proxy which is set at global level.So,even after doing httpurlconnection.openconnection (not httpurlconnection.openconnection (proxy) which takes proxy as parameter) ,the connection is made through the proxy. I would like to know if I can not use the default proxy in cases?
Apologies if the question was not clear.I finally figured it out myself.If proxy is set at system level,httpurlconnection.openconnection() will use the proxy even if we do not pass a proxy object.However to override it we can use -
httpurlconnection.openconnection(Proxy.NO_PROXY)
By passing NO_PROXY the connection will not use proxy for the connection even if it is set as global level.
When I use a SimpleRequestFactory with my AsyncRestTemplate I can easily configure an HTTP proxy server.
I can either do (sample code in Kotlin):
#Bean
open fun asyncRestTemplate(): AsyncRestTemplate {
val proxy = Proxy(Proxy.Type.HTTP, InetSocketAddress("127.0.0.1", 8008))
val requestFactory = SimpleClientHttpRequestFactory().apply {
this.setConnectTimeout(TimeUnit.SECONDS.toMillis(10).toInt())
this.setReadTimeout(TimeUnit.SECONDS.toMillis(10).toInt())
this.setProxy(proxy)
this.setTaskExecutor(taskExecutor())
}
return AsyncRestTemplate(requestFactory)
}
Or I can simply set the corresponding system properties: -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=8008.
However, in the moment that I switch from the SimpleClientHttpRequestFactory to a Netty4ClientHttpRequestFactory there is no evident way to configure the proxy directly and it seems this client does not respect the system properties either.
val requestFactory = Netty4ClientHttpRequestFactory().apply {
this.setConnectTimeout(TimeUnit.SECONDS.toMillis(10).toInt())
this.setReadTimeout(TimeUnit.SECONDS.toMillis(10).toInt())
//this.setProxy(proxy) //???
}
Once I change for the netty client, I have no clue on how to make it go through the proxy.
My interest in using the netty client was that I not only wanted to make async requests, but also I wanted this to be non-blocking. I hope I'm not making a wrong assumption here.
Does anyone know how can I use a proxy server when using the Netty4ClientHttpRequestFactory or perhaps know of an alternative non-blocking client supported by Spring that I could use?
The Netty4ClientHttpRequestFactory (source) and related classes such as Netty4ClientHttpRequest (source) use SimpleChannelInboundHandler for the channel and do not use the proxy handler. Everything is private and unable to be overridden within the source, so there is no way to change it to support Proxies. You would have to almost rewrite the whole thing.
You have other async client options that will work very well and allow you more configuration options. The included Netty one is fairly basic anyway. OkHttpClientHttpRequestFactory and HttpComponentsAsyncClientHttpRequestFactory both let you pass in your own configured client.
To your interest, AsyncRestTemplate's different implementation:
SimpleClientHttpRequestFactory -> simple thread pool, blocking api, proxy supported
OkHttpClient (OkHttp3) -> blocking api, proxy supported
CloseableHttpAsyncClient -> non-blocking nio api, proxy supported
Netty4ClientHttpRequestFactory -> non-blocking nio api, proxy not supported
you can visit https://github.com/wuxudong/VariousAsyncHttpClientPerformance for more details
I am using the ServletTester class provided by Jetty to test one of my servlets.
The servlet reads the the body of the request using InputStream.read() to construct a byte[] which is the decoded and acted on by the servlet.
The ServletTest class provides a method getResponses(ByteArrayBuffer) but I'm unsure how to create one of these in the correct format since it would also need to contain things like headers (e.g. "Content-Type: application/octet-stream).
Can anyone show me an easy way to construct this, preferably using an existing library so that I can use it in a similar way to the HttpTester class.
If there is a "better" way to test servlets (ideally using a local connector rather than via the tcp stack) I'd like to hear that also.
Many thanks,
Why use a mock at all? Why not test the servlet by running it in jetty?
Servlet servlet = new MyServlet();
String mapping = "/foo";
Server server = new Server(0);
Context servletContext = new Context(server, contextPath, Context.SESSIONS);
servletContext.addServlet(new ServletHolder(servlet), mapping);
server.start();
URL url = new URL("http", "localhost", server.getConnectors()[0].getLocalPort(), "/foo?bar");
//get the url...assert what you want
//finally server.stop();
Edit: Just wanting to reassure people that this is very fast. Its also a very reliable indicator of what your code will actually do, because it is in fact doing it.
Spring MVC provides a small set of "mock" classes for the various javax.servlet interfaces, such as HttpServletRequest, HttpSession, and so on. This makes it easy to unit test the likes of a servlet, you just inject mocks into the e.g. doGet() method.
Even if you don't use Spring itself on the server, you can still use the mock from the library, just for your tests.
You can use HttpClient to simplify testing somewhat. Take a look at the following article:
http://roberthanson.blogspot.com/2007/12/testing-servlets-with-junit.html
That in combination with servlet tester should give you what you want unit test wise.