We Keep Coding

How to check if the url is there or not in java?

I am coming to an issue where I need help to check for a url that when I search on a particular code it wont show the url that I have listed. Is there a way to make it work with my code? I tried and created a method below which is generateLink. Thanks for the help.

First, since generateLink will either return a valid URL or null, you need to change this:
jgen.writeStringField("pay_grade_description_link", generateLink(XXX_URL + value.jobClassCd) + ".pdf");
to this:
jgen.writeStringField("pay_grade_description_link", generateLink(value.jobClassCd));
If you concatenate ".pdf" to it, null return values will be meaningless, since null + ".pdf" results in the eight-character string "null.pdf".
Second, you can check the response code of an HttpURLConnection to test a URL’s validity. (In theory, you should be able to use the "OPTIONS" HTTP method to test the URL, but not all servers support it.)
private String generateLink(String jobClassCd) {
String url = XXX_URL + jobClassCd + ".pdf";
try {
HttpURLConnection connection =
(HttpURLConnection) new URL(url).openConnection();
if (connection.getResponseCode() < 400) {
return url;
}
} catch (IOException e) {
Logger.getLogger(JobSerializer.class.getName()).log(Level.FINE,
"URL \"" + url + "\" is not reachable.", e);
}
return null;
}

Android app connecting to shutterstock api throws IOException with Error 401

I am trying to connect my android app to shutterstock api so that it can search for some images there. It uses https scheme + Basic Authentication header to allow users for all search requests. I implemented the functionality in a regular java project using HttpsURLConnection and was able to get correct JSON responses.
The java code looks like this:
HttpsURLConnection urlConnection = (HttpsURLConnection) url.openConnection();//proxy);
String username = "62c01aa824222683004b", password = "dc4ad748a75e4e69ec853ad2435a62b700e66164";
String encoded = Base64.getEncoder().encodeToString((username+":"+password).getBytes("UTF-8"));
System.out.println(encoded.equals("Nj0jMDFhZWE4ZmE4MjY4MzAwNGI6ZGM0YWQ3NDhhNzVlNGU2gWVjODUzYWQ0ZmEzYTYyYjc7MGU2NjE2NA==")); // prints true
urlConnection.setRequestProperty("Authorization", "Basic "+encoded);
When ported this into Android, it was throwing an IOException with 401 error code. As explained in many posts on SO (like the one here), I modified the code accordingly with an extra try-catch as below:
String username = "62c01aa824222683004b", password = "dc4ad748a75e4e69ec853ad2435a62b700e66164", encoded = "";
encoded = Base64.encodeToString((username+":"+password).getBytes("UTF-8"), Base64.URL_SAFE);
Log.e("test", "encoded strings match:" + encoded.equals("Nj0jMDFhZWE4ZmE4MjY4MzAwNGI6ZGM0YWQ3NDhhNzVlNGU2gWVjODUzYWQ0ZmEzYTYyYjc7MGU2NjE2NA==") + "\n" + encoded); // prints false but string is same!!
URL url = new URL(reqUrl);
connection = (HttpsURLConnection) url.openConnection();
connection.setRequestProperty("Authorization", "Basic "+encoded);
try {
if (connection != null) {
connection.connect();
if (200 == connection.getResponseCode()) { // ---> throws IOException
BufferedReader br = new BufferedReader(new InputStreamReader((connection.getInputStream())));
String output;
while ((output = br.readLine()) != null) {
Log.e("test", output);
response.append(output);
}
connection.disconnect();
return response.toString();
}
}
} catch (IOException e) {
try {
Log.e("test", e.getMessage()); // ---> prints "No authentication challenges found"
Log.e("test", connection.getResponseCode() + ":" + connection.getResponseMessage() + connection.getHeaderFields());
//---> prints 401:Unauthorized{null=[HTTP/1.1 401 Unauthorized], cache-control=[no-cache], Connection=[keep-alive], Content-Length=[38], Content-Type=[application/json; charset=utf8], Date=[Tue, 31 May 2016 14:11:28 GMT], Server=[nginx], X-Android-Received-Millis=[1464703888222], X-Android-Sent-Millis=[1464703887592], x-end-user-request-id=[f754ec7f-c344-431b-b641-360aabb70184], x-shutterstock-app-version=[apitwo-625], x-shutterstock-resource=[/v2/images/search]}
if (401 == connection.getResponseCode()) {
InputStream es = connection.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(es));
String output;
while ((output = br.readLine()) != null) {
Log.e("test", output); // ---> prints {"message":"Invalid auth credentials"}
response.append(output);
}
connection.disconnect();
return response.toString();
} else {
Log.e("test","Could not connect! " + connection.getResponseCode() + ":" + connection.getResponseMessage() + ". " + connection.getRequestMethod());
}
}catch (Exception e1){e1.printStackTrace();}
}
I was unable to check the response headers in Firefox's Rest client because it does not send the request to server when I add the Authentication header.
So the questions here are:
Is this the right way to handle the 401 error in Android? Will I get the JSON response in the inner try-catch?
The java program uses exactly the same encoded string as in Android. How come the String.equals() returns "true" in java but "false" in android?
The error message from the server says "Invalid auth credentials". Does the encoded string differ between Android and Java for any reason? If yes, then point 2 makes sense.

I copied the encoded string from the java program into the Android variable and was able to authenticate successfully with shutterstock. So Indeed the encoded strings on Android and Java were different though in UTF-8 format. This also explains the "false" in Android and the "Invalid credentials" message from the server.
Just not sure why/how it differs when both the encoded strings are the same for the human eyes!

Java alternative for curl -T

i must send one text string using java to a IP web cam, before it take picture. So after I read the camera user manual and searched in google, the only thing i found was using cURL. I install it and its run fine, and everything is okay, the text from the file appear in the video streaming. The command is this
curl -T test.xml http://admin:pass#192.168.0.1/Video/inputs/channels/2/overlays/text/2
and the content of test.xml is:
<TextOverlay xmlns="http://www.hikvision.com/ver10/XMLSchema" version="1.0">
<id>2</id>
<enabled>true</enabled>
<posX>5</posX>
<posY>5</posY>
<message>Text here </message>
</TextOverlay>
So I want to send this content using Java, I already tried using post and java.net but I get an error "Server returned HTTP response code: 403 for URL"
Here is my code:
System.out.println("Starting......");
URL url = new URL("http://192.168.0.1/Video/inputs/channels/2/overlays/text/2/");
String data = "<TextOverlay xmlns=\"http://www.hikvision.com/ver10/XMLSchema\" version=\"1.0\">\n"
+ "<id>2</id>\n"
+ "<enabled>true</enabled>\n"
+ "<posX>5</posX>\n"
+ "<posY>5</posY>\n"
+ "<message>Text here</message>\n"
+ "</TextOverlay>";
HttpURLConnection httpConnection = prepareConn(url, null, "admin", "pass");
httpConnection.setDoOutput(true);
httpConnection.setDoInput(true);
httpConnection.setRequestMethod("POST");
httpConnection.setRequestProperty ( "Content-Type", "text/xml" );
OutputStreamWriter out = new OutputStreamWriter(httpConnection.getOutputStream());
out.write(data);
out.flush();
out.close();
System.out.println("Printing......");
System.out.println(httpConnection.getResponseCode());
System.out.println(httpConnection.getResponseMessage());
InputStreamReader reader = new InputStreamReader(httpConnection.getInputStream());
StringBuilder buf = new StringBuilder();
char[] cbuf = new char[2048];
int num;
while(-1 != (num = reader.read(cbuf)))
{
buf.append(cbuf, 0, num);
}
String result = buf.toString();
System.out.println("\nResponse received from server after POST" + result);
}
static private HttpURLConnection prepareConn(final URL url, Properties request_props, String username, String password) throws Error, IOException
{
System.out.println("Authorization......");
if (!url.getProtocol().equalsIgnoreCase("http"))
throw new Error(url.toString() + " is not HTTP!");
final HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setConnectTimeout(300);
conn.setRequestMethod("POST");
final Properties DEFAULT_REQUEST_PROPS = new Properties();
DEFAULT_REQUEST_PROPS.setProperty("charset", "utf-8");
final Properties props = new Properties(DEFAULT_REQUEST_PROPS);
if (request_props != null)
for (final String name : request_props.stringPropertyNames())
props.setProperty(name, request_props.getProperty(name));
for (final String name : props.stringPropertyNames())
conn.setRequestProperty(name, props.getProperty(name));
if(null != username && null != password)
conn.setRequestProperty("Authorization", "Basic " + new BASE64Encoder().encode((username+":"+password).getBytes()));
return conn;
}
Hope someone can help :)
All the best !

I just use wrong RequestMethod, after deep research I found that i must use PUT not POST request. Now just change setRequestMethod("POST") to setRequestMethod("PUT") and works like a charm.

Display captcha in Jframe

I want to create application for registration in Java, then I want to submit the information to a website. This is only experiment so the information for the registration(e.g username, password) will be submit with GET request. However I want to integrate captcha with the registration and I want to display it on the Jframe and submit the answer along side with the other data. I have no idea how to get the captcha image, and then submit the data. Also I think to use the new reCaptcha(where it ask you to select foods). Any ideas how to do this?
Edit:
I know how to display the image with JLabel, I also was able to find a way to extract it Get image for captcha session .Now i'm wondering how to send the response.

To send a response you will probably need a Session ID from the server and the clients answer then just send a GET request to the server with both of the values
public void getMethod() throws IOException
{
String userAgent = "Java/" + Runtime.class.getPackage().getImplementationVersion();
//The server will need to know what "question" we are answering so it sent us the captha and a sesion ID
//example is just a random one you will need to figure out how to get a sesion id
String captchaSesionParam = "captchaSesionID=";
String captchaSesionID = UUID.randomUUID().toString();
//user has completed captha client side here is their answer
String queryParam = "answer=";
String answer = "blah blah answer";
String urlString = "https://127.0.0.1/?" + queryParam + URLEncoder.encode(answer, "UTF-8") + "&" + captchaSesionParam + URLEncoder.encode(captchaSesionID, "UTF-8");
URL url = new URL(urlString);
//Open a HTTPS connection to the URL
HttpURLConnection con = (HttpURLConnection) url.openConnection();
//set request method to GET
con.setRequestMethod("GET");
//set user agent to our agent (by default I believe that this is 'Java/Version')
con.setRequestProperty("User-Agent", userAgent);
//print out debug info about request method and url
System.out.println(con.getRequestMethod() + " URL : " + url);
int responseCode = con.getResponseCode();
System.out.println("Server response code: " + responseCode);
try (BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream(), "UTF-8"));)
{
String line;
List<String> lines = new ArrayList<String>();
while ((line = in.readLine()) != null)
{
lines.add(line);
}
//parse lines received from server to see if the captcha was (in)correct
//print lines for debug
for(String l : lines)
{
System.out.println(l);
}
}
}

java inputstream print to console the content

sock = new Socket("www.google.com", 80);
out = new BufferedOutputStream(sock.getOutputStream());
in = new BufferedInputStream(sock.getInputStream());
When i try to do printing out of content inside "in" like below
BufferedInputStream bin = new BufferedInputStream(in);
int b;
while ( ( b = bin.read() ) != -1 )
{
char c = (char)b;
System.err.print(""+(char)b); //This prints out content that is unreadable.
//Isn't it supposed to print out html tag?
}

If you want to print the content of a web page, you need to work with the HTTP protocol. You do not have to implement it yourself, the best way is to use existing implementations such as the java API HttpURLConnection or Apache's HttpClient
Here is an example of how to do it with HttpURLConnection:
URL url = new URL("http","www.google.com");
HttpURLConnection urlc = (HttpURLConnection)url.openConnection();
urlc.setAllowUserInteraction( false );
urlc.setDoInput( true );
urlc.setDoOutput( false );
urlc.setUseCaches( true );
urlc.setRequestMethod("GET");
urlc.connect();
// check you have received an status code 200 to indicate OK
// get the encoding from the Content-Type header
BufferedReader in = new BufferedReader(new InputStreamReader(urlc.getInputStream()));
String line = null;
while((line = in.readLine()) != null) {
System.out.println(line);
}
// close sockets, handle errors, etc.
As written above, you can save traffic by adding the Accept-Encoding header and check the
Content-Encoding header of the response.
Here is an HttpClient Example, taken from here:
// Create an instance of HttpClient.
HttpClient client = new HttpClient();
// Create a method instance.
GetMethod method = new GetMethod(url);
// Provide custom retry handler is necessary
method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(3, false));
try {
// Execute the method.
int statusCode = client.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: " + method.getStatusLine());
}
// Read the response body.
byte[] responseBody = method.getResponseBody();
// Deal with the response.
// Use caution: ensure correct character encoding and is not binary data
System.out.println(new String(responseBody));
} catch (HttpException e) {
System.err.println("Fatal protocol violation: " + e.getMessage());
e.printStackTrace();
} catch (IOException e) {
System.err.println("Fatal transport error: " + e.getMessage());
e.printStackTrace();
} finally {
// Release the connection.
method.releaseConnection();
}

Very easy to create a String from a Stream using Java 8 Stream API:
new BufferedReader(new InputStreamReader(in)).lines().collect(Collectors.joining("\n"))
Using IntelliJ I even can set this beeing a debug expression:
I guess in Eclipse it will work similar.

If you what to fetch the content of a webpage, you should take a look at apache httpclient instead of coding this yourself, expect for learning purposes or any other really good reason.