We Keep Coding

Read from URL if content is image [duplicate]

I am trying to write a java program that will automatically download and name some of my favorite web comics. Since I will be requesting multiple objects from the same domain, I wanted to have a persistent http connection that I could keep open until all the comics have been downloaded. Below is my work-in-progress. How do I make another request from the same domain but different path without opening a new http connection?
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
public class ComicDownloader
{
public static void main(String[] args)
{
URL url = null;
HttpURLConnection httpc = null;
BufferedReader input = null;
try
{
url = new URL("http://www.cad-comic.com/cad/archive/2002");
httpc = (HttpURLConnection) url.openConnection();
input = new BufferedReader(new InputStreamReader(httpc.getInputStream()));
String inputLine;
while ((inputLine = input.readLine()) != null)
{
System.out.println(inputLine);
}
input.close();
httpc.disconnect();
}
catch (IOException ex)
{
System.out.println(ex);
}
}
}

According to the documentation here, HTTP persistence is being handled transparently in Java, although it gives you the options to control it too via http.keepAlive and http.maxConnections system properties.
However,
The current implementation doesn't
buffer the response body. Which means
that the application has to finish
reading the response body or call
close() to abandon the rest of the
response body, in order for that
connection to be reused. Furthermore,
current implementation will not try
block-reading when cleaning up the
connection, meaning if the whole
response body is not available, the
connection will not be reused.
Take a look at the link and see if it really helps you.

According to this link http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html, HTTP connection reuse is enabled by default, you can use Wireshark to check the interactions between your client and server. The first request contains TCP and SSL handshakes(if your request is https), the subsequent requests fired in the keep-alive time, contains no TCP and SSL handshakes, just application data transfers.

Even though HttpURLConnection enable keep-alive by default, it is not guaranteed that HttpURLConnection uses same TCP connection for multiple HTTP requests. I faced same kind of issue when writing HTTPS client application. Solved this issue by using single instance of SSLContext, SSLSocketFactory and HttpsURLConnection.
public class MyHTTPClient {
private SSLContext mSSLContext = null;
private SSLSocketFactory mSSLSocketFactory = null;
private HttpsURLConnection mConnection = null;
public void init() {
//Setup SSL context and Socket factory here
}
pubblic void sendRequest() {
URL url = new URL("https://example.com/request_receiver");
mConnection = (HttpsURLConnection) url.openConnection();
mConnection.setSSLSocketFactory(mSSLSocketFactory);
// Setup request property and send request
// Open input stream to read response
// Close output, input streams
mConnection.disconnect();
}
}

Kerberos authentication for web request in Java?

I'm trying to implement exactly this: https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/lab/part6.html
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.Authenticator;
import java.net.PasswordAuthentication;
import java.net.URL;
public class RunHttpSpnego {
static final String kuser = "username"; // your account name
static final String kpass = "password"; // your password for the account
static class MyAuthenticator extends Authenticator {
public PasswordAuthentication getPasswordAuthentication() {
// I haven't checked getRequestingScheme() here, since for NTLM
// and Negotiate, the usrname and password are all the same.
System.err.println("Feeding username and password for "
+ getRequestingScheme());
return (new PasswordAuthentication(kuser, kpass.toCharArray()));
}
}
public static void main(String[] args) throws Exception {
Authenticator.setDefault(new MyAuthenticator());
URL url = new URL(args[0]);
InputStream ins = url.openConnection().getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(ins));
String str;
while((str = reader.readLine()) != null)
System.out.println(str);
}
}
I've already reduced to trying just to execute the sample code snippet, but I don't know how to overcome this issue I'm facing.
As I understand, the flow for such requests should look like, as detailed in this document as well:
1. The clients sends a HTTP GET request to the server
2. The server responds with a HTTP 401, and a WWW Authenticate: Negotiate header
3. Then the client sends another HTTP GET request, this time with the proper Negotiate header in the request.
My problem is, when I try to run that sample, the request fails with an IOException obviously, because of the server responding with a 401, and then I don't know how to issue the second request with the header. I was trying to add a header as with Basic Authentication, but in that case I don't know where to get the Negotiate value for that from.
I did a packet capture, I can see the TGS ticket being requested from the KDC, so I think that part is OK, I just don't know how it should be applied to my request. I did the same packet capture with other scripts connecting to the same web service, and I can see the exact same flow what I've described above.
So, to summarize my question: how should I add the authentication header to my request? Do I need to do it explicitely in my code, if so, how do I do it?

JBPM Console Remote authetication through java

I have deployed jbpm-cosole on my localhost, and I have created a java application to make REST calls for process transaction. So, my question is.
As jbpm-console uses spring-security, how to send authentication request to JBPM?
After successful authentication I want to store authentication token as a java object to make further transactions without login.
or is there any another way to do that please let me know.

This is a sample code which returns the tasks list of a user. You just need to set the authentication details in request header as shown in this code. This works with jbpm-6.1.0.Final.
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.util.EntityUtils;
public class JBPMRest {
public static void getTaskSummaryList() throws Exception {
String status = "Reserved";
String actor = "krisv";
String addr = "http://localhost:8080/jbpm-console/rest/task/query?status=" + status + "&potentialOwner=" + actor;
try {
HttpClient client = HttpClientBuilder.create().build();
HttpGet get = new HttpGet(addr);
String authData = "krisv" + ":" + "krisv";
String encoded = new sun.misc.BASE64Encoder().encode(authData.getBytes());
get.setHeader("Content-Type", "application/json");
get.setHeader("Authorization", "Basic " + encoded);
get.setHeader("ACCEPT", "application/xml");
HttpResponse cgResponse = client.execute(get);
String content = EntityUtils.toString(cgResponse.getEntity());
System.out.println(content);
} catch (Exception e) {
throw new Exception("Error consuming service.", e);
}
}
}

You can send basic authentication information (username + password) in the header of your REST call. The remote Java client implementation for example does this as well. If you use the remote Java client, you can reuse the same engine for subsequent calls as well.

URLConnection in java using username, password, token

I am trying to Connect to url using URLConnection, in java with username and password.
This is the following code I am using:
package com.nivi.org.client;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import com.sun.jersey.core.util.Base64;
public class GetURLContent {
public static void main(String[] args) {
URL url;
try {
url = new URL("http://sampleurl.co.uk");
URLConnection conn = url.openConnection();
String username = "username";
String password = "password";
String Token = "zzzzzzzzzzzzzzzzzzzzz";
System.setProperty("http.proxyHost", "proxyhostname");
System.setProperty("http.proxyPort", "8080");
String userpass = username + ":" + password;
String basicAuth = "Basic " + new String(new Base64().encode(userpass.getBytes()));
conn.setRequestProperty ("Authorization", basicAuth);
conn.setRequestProperty ("Token", Token);
BufferedReader br = new BufferedReader(
new InputStreamReader(conn.getInputStream()));
System.out.println("br............."+br.toString());
br.close();
System.out.println("Done");
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
=======================================
I am getting the following error
java.io.IOException: Server returned HTTP response code: 403 for URL:
http://sampleurl.co.uk
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
at com.nivi.org.client.GetURLContent.main(GetURLContent.java:63)
=======================================
First of all my question is
Are these following lines in the code are correct?
conn.setRequestProperty ("Authorization", basicAuth);
conn.setRequestProperty ("Token", Token);

Are these following lines in the code are correct?
It depends on how the actual website you are accessing implements its user authentication.
What you appear to be doing is a combination of HTTP Basic Authentication (i.e. the Authorization header), and something involving a non-standard header called Token. This may be sufficient, if the website supports Basic Authentication.
You should probably read the website's programmer documentation of how their web APIs work. If there isn't any such documentation available to you, use your browsers web development tools to identify the mechanism that the site is using when you log in via a web browser ... and attempt to get your client to behave the same way.
One thing to note is that sending a Basic authorization response in an HTTP request is insecure. Use an HTTPS request if that is an option.

Connection to a URL from within an applet using Apache's HttpClient vs using the JDK's URLConnection

In the following code, I have verified that connecting to a URL from within an applet preserves the browser's session if JDK's URLConnection class is used. However, this is not the case if Apache's HttpClient library is used. Does anyone know why? Alternatively, is there a way for me to set the connection instance to be used by an HttpClient instance?
import java.applet.Applet;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URISyntaxException;
import java.net.URL;
import javax.net.ssl.SSLException;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.DefaultHttpClient;
public class HttpClientTesterApplet extends Applet {
private static final long serialVersionUID = -1599714556710568947L;
public void testHttpClient() throws ClientProtocolException, IOException,
URISyntaxException {
URL url = new URL(String.format("%s://localhost:%s/%s/testHttpClient",
getParameter("protocol"), getParameter("port"),
getParameter("context")));
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(url.toURI());
System.out.println("Executing request " + post.getURI());
try {
System.out
.println(client.execute(post, new BasicResponseHandler()));
} catch (SSLException e) {
System.out.println(e.getMessage());
}
System.out.println("Executed request " + post.getURI());
System.out.println("Opening connection " + url);
HttpURLConnection urlConnection = (HttpURLConnection) url
.openConnection();
System.out.println("Opened connection " + url);
urlConnection.setRequestMethod("POST");
System.out.println("Connecting");
urlConnection.connect();
System.out.println("Connected");
InputStream inputStream = urlConnection.getInputStream();
try {
while (inputStream.read() != -1) {
System.out.println("Reading");
}
} finally {
inputStream.close();
}
}
}

This is a common problem with libraries implementing their own URL connection via Socket. Apparently, the JRE implementation of the URLConnection class can get to the browser information directly. We had to employ the technique as mentioned by oscargm above, i.e. on the appserver writing the request cookies to be the parameters to the applet AND getting to the browser's document cookies using JavaScript (this is for the case of SSO where the set of cookies may not be the same because of the intermediate agent -- proxy servers). Note that if the cookies are HttpOnly -- the javascript code will fail.

You must send the jsessionid cookie or rewrite your URL to use the jsessionid.
That's the way the server knows your session.
If you generate the applet tag in a JSP page dynamically you can pass the jsessionidvalue to the applet as a parameter and then use it.
post.setHeader("Cookie", "jsessionid=" + jsessionidValue );

I think that you're using an older version of HttpClient. Check out HttpClient's website.
In the current API, you can use HttpState in the execute method, so that your code could look like this:
HttpClient client = new HttpClient();
HttpMethod method = new PostMethod(url.toURI());
HttpState state = new HttpState();
client.executeMethod(HttpConfiguration.ANY_HOST_CONFIGURATION, method, state);
In the next execution, pass the same "state" object, and you'll get the credentials and cookies preserved.

Possible causes, is that you have not done a disconnect() when using URLConnection, however, the apache library will close the connection when you are done with it.

Thi is an important issue.
The standard java.net.URLConnection class integrates seamlessly with the java plugin and the web browser, can inherit session, HTTP authentication tokens, proxy connectors, etc.
The guys at Apache Commons made a gross mistake when they decided to implement HttpClient from Socket (ie, from scratch) instead of just developing on top of the standard java.net.URL* classes. HttpClient does not inherit from java.net.URLConnection so it cannot inherit its advanced enterprise features.
Maybe OpenSource projects are not so smart as they think.

I could make it work without passing cookies as arguments from the Web Page with this code:
private String retrieveCookies(URL url) throws IOException, URISyntaxException
{
String cookieValue = null;
CookieHandler handler = CookieHandler.getDefault();
if (handler != null) {
Map<String, List<String>> headers = handler.get(url.toURI(), new HashMap<String, List<String>>());
List<String> cookiesList = headers.get("Cookie");
if (cookiesList != null)
{
for (String v : cookiesList) {
if (cookieValue == null)
cookieValue = v;
else
cookieValue = cookieValue + ";" + v;
}
}
}
return cookieValue;
}
...
httppost.addHeader("Cookie", retrieveCookies(new URL(uploadUrl)));
JDK's class CookieHandler can fortunately get the cookies from the "system" store. In this case it's the browser store, accesed via the Java Plugin.
Sort of "manual work", but it works.
NOTE: I found the code here