I am trying to write a java program that will automatically download and name some of my favorite web comics. Since I will be requesting multiple objects from the same domain, I wanted to have a persistent http connection that I could keep open until all the comics have been downloaded. Below is my work-in-progress. How do I make another request from the same domain but different path without opening a new http connection?
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
public class ComicDownloader
{
public static void main(String[] args)
{
URL url = null;
HttpURLConnection httpc = null;
BufferedReader input = null;
try
{
url = new URL("http://www.cad-comic.com/cad/archive/2002");
httpc = (HttpURLConnection) url.openConnection();
input = new BufferedReader(new InputStreamReader(httpc.getInputStream()));
String inputLine;
while ((inputLine = input.readLine()) != null)
{
System.out.println(inputLine);
}
input.close();
httpc.disconnect();
}
catch (IOException ex)
{
System.out.println(ex);
}
}
}
According to the documentation here, HTTP persistence is being handled transparently in Java, although it gives you the options to control it too via http.keepAlive and http.maxConnections system properties.
However,
The current implementation doesn't
buffer the response body. Which means
that the application has to finish
reading the response body or call
close() to abandon the rest of the
response body, in order for that
connection to be reused. Furthermore,
current implementation will not try
block-reading when cleaning up the
connection, meaning if the whole
response body is not available, the
connection will not be reused.
Take a look at the link and see if it really helps you.
According to this link http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html, HTTP connection reuse is enabled by default, you can use Wireshark to check the interactions between your client and server. The first request contains TCP and SSL handshakes(if your request is https), the subsequent requests fired in the keep-alive time, contains no TCP and SSL handshakes, just application data transfers.
Even though HttpURLConnection enable keep-alive by default, it is not guaranteed that HttpURLConnection uses same TCP connection for multiple HTTP requests. I faced same kind of issue when writing HTTPS client application. Solved this issue by using single instance of SSLContext, SSLSocketFactory and HttpsURLConnection.
public class MyHTTPClient {
private SSLContext mSSLContext = null;
private SSLSocketFactory mSSLSocketFactory = null;
private HttpsURLConnection mConnection = null;
public void init() {
//Setup SSL context and Socket factory here
}
pubblic void sendRequest() {
URL url = new URL("https://example.com/request_receiver");
mConnection = (HttpsURLConnection) url.openConnection();
mConnection.setSSLSocketFactory(mSSLSocketFactory);
// Setup request property and send request
// Open input stream to read response
// Close output, input streams
mConnection.disconnect();
}
}
Related
I'm trying to implement exactly this: https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/lab/part6.html
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.Authenticator;
import java.net.PasswordAuthentication;
import java.net.URL;
public class RunHttpSpnego {
static final String kuser = "username"; // your account name
static final String kpass = "password"; // your password for the account
static class MyAuthenticator extends Authenticator {
public PasswordAuthentication getPasswordAuthentication() {
// I haven't checked getRequestingScheme() here, since for NTLM
// and Negotiate, the usrname and password are all the same.
System.err.println("Feeding username and password for "
+ getRequestingScheme());
return (new PasswordAuthentication(kuser, kpass.toCharArray()));
}
}
public static void main(String[] args) throws Exception {
Authenticator.setDefault(new MyAuthenticator());
URL url = new URL(args[0]);
InputStream ins = url.openConnection().getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(ins));
String str;
while((str = reader.readLine()) != null)
System.out.println(str);
}
}
I've already reduced to trying just to execute the sample code snippet, but I don't know how to overcome this issue I'm facing.
As I understand, the flow for such requests should look like, as detailed in this document as well:
1. The clients sends a HTTP GET request to the server
2. The server responds with a HTTP 401, and a WWW Authenticate: Negotiate header
3. Then the client sends another HTTP GET request, this time with the proper Negotiate header in the request.
My problem is, when I try to run that sample, the request fails with an IOException obviously, because of the server responding with a 401, and then I don't know how to issue the second request with the header. I was trying to add a header as with Basic Authentication, but in that case I don't know where to get the Negotiate value for that from.
I did a packet capture, I can see the TGS ticket being requested from the KDC, so I think that part is OK, I just don't know how it should be applied to my request. I did the same packet capture with other scripts connecting to the same web service, and I can see the exact same flow what I've described above.
So, to summarize my question: how should I add the authentication header to my request? Do I need to do it explicitely in my code, if so, how do I do it?
I try to send a SOAP message in an XML file to a webservice and than grab the binary output and decode it. Endpoint uses HTTPS protocol, so I used TrustManager in my code to avoid PKIX problems. You can see my code here:
import javax.net.ssl.*;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.security.cert.X509Certificate;
public class Main{
public static void sendSoapRequest() throws Exception {
String SOAPUrl = "URL HERE";
String xmlFile2Send = ".\\src\\request.xml";
String responseFileName = ".\\src\\response.xml";
String inputLine;
TrustManager[] trustAllCerts = new TrustManager[] { new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() { return null; }
public void checkClientTrusted(X509Certificate[] certs, String authType) { }
public void checkServerTrusted(X509Certificate[] certs, String authType) { }
} };
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
// Create all-trusting host name verifier
HostnameVerifier allHostsValid = new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) { return true; }
};
// Install the all-trusting host verifier
HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);
// Create the connection with http
URL url = new URL(SOAPUrl);
URLConnection connection = url.openConnection();
HttpURLConnection httpConn = (HttpURLConnection) connection;
FileInputStream fin = new FileInputStream(xmlFile2Send);
ByteArrayOutputStream bout = new ByteArrayOutputStream();
copy(fin, bout);
fin.close();
byte[] b = bout.toByteArray();
StringBuffer buf=new StringBuffer();
String s=new String(b);
b=s.getBytes();
// Set the appropriate HTTP parameters.
httpConn.setRequestProperty("Content-Length", String.valueOf(b.length));
httpConn.setRequestProperty("Content-Type", "text/xml; charset=utf-8");
httpConn.setRequestProperty("SOAPAction", "");
httpConn.setRequestMethod("POST");
httpConn.setDoOutput(true);
OutputStream out = httpConn.getOutputStream();
out.write(b);
out.close();
// Read the response.
httpConn.connect();
System.out.println("http connection status :"+ httpConn.getResponseMessage());
InputStreamReader isr = new InputStreamReader(httpConn.getInputStream());
BufferedReader in = new BufferedReader(isr);
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
FileOutputStream fos=new FileOutputStream(responseFileName);
copy(httpConn.getInputStream(),fos);
in.close();
}
public static void copy(InputStream in, OutputStream out) throws IOException {
synchronized (in) {
synchronized (out) {
byte[] buffer = new byte[256];
while (true) {
int bytesRead = in.read(buffer);
if (bytesRead == -1)
break;
out.write(buffer, 0, bytesRead);
}
}
}
}
public static void main(String args[]) throws Exception {
sendSoapRequest();
}
}
I get following error code, when I execute this.
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 403 for URL
Your implementation is alright, the problem is related to your Content-Type header, in fact.
The value text/xml; charset=utf-8 is the default Content-Type of SOAP 1.1, which is probably not the version of yours. SOAP 1.2 expects a header of type application/soap+xml; charset=utf-8, so changing your line of code to this one below is gonna make it working:
httpConn.setRequestProperty("Content-Type", "application/soap+xml; charset=utf-8");
In SoapUI, it's possible to check the headers calling the request and going to the Headers tab on the bottom of the window:
Then, you can compare the differences between your application configs and the SoapUI ones.
403 error might be related to your soap request headers being sent to the server.
All Hosts valid will allow your Java App to trust the SSL Cert for the URL.
Check if your server is expecting soap header with username/password. If you have access to this server, you can check through the web server logs on where your request is failing. Error code points to to missing Soap Header particularly Soap Headers with username and password
Wonder if your SOAP request contains any kind of authentication information in headers like SAML. One option is, in your above code where you read the file and send the data to server, instead of sending it to server you dump it to another file. Dump that byteoutputstream. Then copy text from that file and put it in SOAP UI and try running that. Does that work?
In a similar situation we have been some time before, and as long as trying TrustManager didn't work as expected, we managed to overcome this problem by installing the certificate from server to JVM's keystore (JVM used to run the application). More information about how to do it you can find in several posts, like
How to import a .cer certificate into a java keystore?
I am aware that it is a try to force JVM to accept SSL certificates, and this functionality would be better to live in application context, but as long as we were building a web application which ran in specific application servers, the implemented solution was an accepted one.
I have configure IIS 8.0 with PHP 5.3 , along with SQL Server.
I am not sure what is going wrong. I am able to see my JSON string in web browser , when I type in URL in the browser,
But When I passed it to the code below, it is return me
IIS 8.0 Detailed Error - 401.2 - Unauthorized
401.2 Error page in my eclipse.
I am connecting to the database via Windows Authentication. Therefore , I have set Windows Authentication enabled in my IIS. I cannot able any other authentication other than Windows authentication , then my JSONstring is not return at all from the browser as well., in that case.
I have tried the solution mentioned in the below URL
http://support.microsoft.com/kb/942043
The problem still persists
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.params.CoreProtocolPNames;
public class JSONTester {
/**
* #param args
* #throws Exception
*/
public static void main(String[] args)
{
InputStream is = null;
String json = "";
try
{
// defaultHttpClient
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost("http://localhost/hive/get_all_products.php");
httpPost.getParams().setBooleanParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE, false);
HttpResponse httpResponse = httpClient.execute(httpPost);
HttpEntity httpEntity = httpResponse.getEntity();
is = httpEntity.getContent();
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(
is, "iso-8859-1"), 8);
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
json = sb.toString();
} catch (Exception e) {
System.out.println(e.toString());
}
System.out.println(json);
}
}
Any help would be great .
Thanks !!!
Internet Explorer is the only browser that does the NTLM authentication using integrated mode out-of-the-box, this is why your URL works in your 'browser'. Your Java HttpClient must authenticate explicitly and answer the NTLM challenge from the server. See HttpClient NTLM Authentication:
NTLM is the most complex of the authentication protocols supported by
HttpClient. It is a proprietary protocol designed by Microsoft with no
publicly available specification... NTLM
authentication requires an instance of NTCredentials be available for
the domain name of the server or the default credentials. Note that
since NTLM does not use the notion of realms HttpClient uses the
domain name of the server as the name of the realm. Also note that the
username provided to the NTCredentials should not be prefixed with the
domain - ie: "adrian" is correct whereas "DOMAIN\adrian" is not
correct...
HttpClient provides limited support for what is known as NTLMv1, the early version of the NTLM protocol. HttpClient does not support NTLMv2 at all.
The important thing to take home is that HttpClient does not support integrated authentication. You must provide explicit credentials, which makes it always probe to bad practices because of the requirement to provide the account password explicitly. Make sure you do not store it but ask it from the user, much the same way Firefox does when visiting a site that requests an NTLM challenge.
Read The NTLM Authentication Protocol and Security Support Provider for more details.
Make sure the account you are authenticating with has rights to the files in the website directory.
I'm trying to create a http server in Java which is capable of providing keep-alive connections. I'm using the com.sun.net.httpserver.HttpServer class.
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import com.sun.net.httpserver.Headers;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
public class httpHandler implements HttpHandler {
private String resp = "<?xml version='1.0'?><root-node></root-node>";
private OutputStream os = null;
public void handle(HttpExchange t) throws IOException {
System.out.println("Handling message...");
java.io.InputStream is = t.getRequestBody();
System.out.println("Got request body. Reading request body...");
byte[] b = new byte[500];
is.read(b);
System.out.println("This is the request: " + new String(b));
String response = resp;
Headers header = t.getResponseHeaders();
header.add("Connection", "Keep-Alive");
header.add("Keep-Alive", "timeout=14 max=100");
header.add("Content-Type", "application/soap+xml");
t.sendResponseHeaders(200, response.length());
if(os == null) {
os = t.getResponseBody();
}
os.write(response.getBytes());
System.out.println("Done with exchange. Closing connection");
os.close();
}
public static void main(String[] args) {
HttpServer server = null;
try {
server = HttpServer.create(new InetSocketAddress(8080), 5);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
server.createContext("/", new httpHandler());
server.setExecutor(null); // creates a default executor
System.out.println("Starting server...");
server.start();
}
}
The client does not close the connection. The server seems to close it instead directly after the exchange has occurred. I tried deleting the os.close line but then the server will not reply to the second request. But it doesn't close it either. I have a feeling it involves doing something in the main code with the server object but I have no idea what. Google isn't turning up much either.
Anyone here got any ideas? Any help would be much appreciated.
It looks like the problem is you didn't drain all data from request. You should keep doing is.read() until it returns -1, then close it.
Since you didn't drain the request, there are still bytes left. The server cannot simply "jump" to the next request; it's not like a disk, more like a tape. Server has to read(and discard) all data from current request, before the it can reach the next request.
Without a limit this can be used to attack the server; so the server would only attempt to drain up to a limit, which is 64K by default. You are probably receiving a request bigger than 64K.
Usually the handler should read the entire request first. Otherwise how does it know how to serve the request?
More seriously, if the request is not drained first, deadlock can happen. Clients are typically simple: they writes the request, then read the response. If the server writes the response before it reads all the request, the client could be still writing the request. Both are writing to each other but neither is reading. If buffers are full then we are in a deadlock, both are blocked on write. Note there's no timeout for write()!
Why do you get the ResponseBody OutputStream only if the previous was null?
if(os == null) {
os = t.getResponseBody();
}
Might be better to get the OutputStream each time as you cannot be sure if the previous one is identical to the one of current Request.
try to call t.close(), the server will reply
It's the client that sends the keep-alive headers. If it does that, HttpServer doesn't close the connection. You don't have to do anything about it.
In the following code, I have verified that connecting to a URL from within an applet preserves the browser's session if JDK's URLConnection class is used. However, this is not the case if Apache's HttpClient library is used. Does anyone know why? Alternatively, is there a way for me to set the connection instance to be used by an HttpClient instance?
import java.applet.Applet;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URISyntaxException;
import java.net.URL;
import javax.net.ssl.SSLException;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.DefaultHttpClient;
public class HttpClientTesterApplet extends Applet {
private static final long serialVersionUID = -1599714556710568947L;
public void testHttpClient() throws ClientProtocolException, IOException,
URISyntaxException {
URL url = new URL(String.format("%s://localhost:%s/%s/testHttpClient",
getParameter("protocol"), getParameter("port"),
getParameter("context")));
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(url.toURI());
System.out.println("Executing request " + post.getURI());
try {
System.out
.println(client.execute(post, new BasicResponseHandler()));
} catch (SSLException e) {
System.out.println(e.getMessage());
}
System.out.println("Executed request " + post.getURI());
System.out.println("Opening connection " + url);
HttpURLConnection urlConnection = (HttpURLConnection) url
.openConnection();
System.out.println("Opened connection " + url);
urlConnection.setRequestMethod("POST");
System.out.println("Connecting");
urlConnection.connect();
System.out.println("Connected");
InputStream inputStream = urlConnection.getInputStream();
try {
while (inputStream.read() != -1) {
System.out.println("Reading");
}
} finally {
inputStream.close();
}
}
}
This is a common problem with libraries implementing their own URL connection via Socket. Apparently, the JRE implementation of the URLConnection class can get to the browser information directly. We had to employ the technique as mentioned by oscargm above, i.e. on the appserver writing the request cookies to be the parameters to the applet AND getting to the browser's document cookies using JavaScript (this is for the case of SSO where the set of cookies may not be the same because of the intermediate agent -- proxy servers). Note that if the cookies are HttpOnly -- the javascript code will fail.
You must send the jsessionid cookie or rewrite your URL to use the jsessionid.
That's the way the server knows your session.
If you generate the applet tag in a JSP page dynamically you can pass the jsessionidvalue to the applet as a parameter and then use it.
post.setHeader("Cookie", "jsessionid=" + jsessionidValue );
I think that you're using an older version of HttpClient. Check out HttpClient's website.
In the current API, you can use HttpState in the execute method, so that your code could look like this:
HttpClient client = new HttpClient();
HttpMethod method = new PostMethod(url.toURI());
HttpState state = new HttpState();
client.executeMethod(HttpConfiguration.ANY_HOST_CONFIGURATION, method, state);
In the next execution, pass the same "state" object, and you'll get the credentials and cookies preserved.
Possible causes, is that you have not done a disconnect() when using URLConnection, however, the apache library will close the connection when you are done with it.
Thi is an important issue.
The standard java.net.URLConnection class integrates seamlessly with the java plugin and the web browser, can inherit session, HTTP authentication tokens, proxy connectors, etc.
The guys at Apache Commons made a gross mistake when they decided to implement HttpClient from Socket (ie, from scratch) instead of just developing on top of the standard java.net.URL* classes. HttpClient does not inherit from java.net.URLConnection so it cannot inherit its advanced enterprise features.
Maybe OpenSource projects are not so smart as they think.
I could make it work without passing cookies as arguments from the Web Page with this code:
private String retrieveCookies(URL url) throws IOException, URISyntaxException
{
String cookieValue = null;
CookieHandler handler = CookieHandler.getDefault();
if (handler != null) {
Map<String, List<String>> headers = handler.get(url.toURI(), new HashMap<String, List<String>>());
List<String> cookiesList = headers.get("Cookie");
if (cookiesList != null)
{
for (String v : cookiesList) {
if (cookieValue == null)
cookieValue = v;
else
cookieValue = cookieValue + ";" + v;
}
}
}
return cookieValue;
}
...
httppost.addHeader("Cookie", retrieveCookies(new URL(uploadUrl)));
JDK's class CookieHandler can fortunately get the cookies from the "system" store. In this case it's the browser store, accesed via the Java Plugin.
Sort of "manual work", but it works.
NOTE: I found the code here