We Keep Coding

Why am I not getting a MalformedURLException on incorrect URL's?

Here's my code, I'm wondering why I'm not getting a MalformedURLException on this, on wrong URL's it just prints Site is Down, when it should be Incorrect URL. Maybe I'm misunderstanding the exception, for example, I thought https://goofdggle.djaosi a malformed URL, but my thing is returning Site is Down when it should be Incorrect URL. This is a Spring Boot app by the way.
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
#RestController
public class URLCheckController {
private String siteup = "Site is up!";
private String sitedown = "Site is down";
private String incorrectURL = "Incorrect URL!";
#GetMapping("/check")
public String getURLStatusMessage(#RequestParam String url) {
String returnMessage = "";
try {
URL urlObj = new URL(url);
HttpURLConnection conn = (HttpURLConnection) urlObj.openConnection();
conn.setRequestMethod("GET");
conn.connect();
int responseCategory = conn.getResponseCode() / 100;
if(responseCategory != 2 && responseCategory != 3) {
returnMessage = sitedown;
} else {
returnMessage = siteup;
}
} catch (MalformedURLException e) {
returnMessage = incorrectURL;
} catch (IOException e) {
returnMessage = sitedown;
}
return returnMessage;
}
}

Send automatic message on a given date

I have made a SMS application with Java that works like a charm. I want the application to send SMS on a given date. I am using the Quartz Job Scheduling for that. First I need these two applications to be connected to each other. Then I need Quartz to collect a date and time from a list (can be excel list) and send a message. Is this possible? Appreciate any help.
Here is the message application:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import org.apache.commons.codec.binary.Base64;
public class SMSApplication {
public static void main(String[] args) {
try {
String phoneNumber = "+XXXXXXXXX";
String appKey = "XXXXXXX";
String appSecret = "XXXXXXXX";
String message = "Hello world!";
URL url = new URL("https://messagingapi.sinch.com/v1/sms/" + phoneNumber);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/json");
String userCredentials = "application\\" + appKey + ":" + appSecret;
byte[] encoded = Base64.encodeBase64(userCredentials.getBytes());
String basicAuth = "Basic " + new String(encoded);
connection.setRequestProperty("Authorization", basicAuth);
String postData = "{\"Message\":\"" + message + "\"}";
OutputStream os = connection.getOutputStream();
os.write(postData.getBytes());
StringBuilder response = new StringBuilder();
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line;
while ( (line = br.readLine()) != null)
response.append(line);
br.close();
os.close();
System.out.println(response.toString());
} catch (IOException e) {
e.printStackTrace();
}
}
}
Here is the Quartz job class:
import java.util.Date;
import org.quartz.Job;
import org.quartz.JobExecutionContext;
import org.quartz.JobExecutionException;
public class QuartzJob implements Job{
#Override
public void execute(JobExecutionContext jec) throws JobExecutionException {
System.out.println("Hello");
System.out.println(new Date())
}
}
And here is the Quartz main class:
import org.quartz.JobBuilder;
import org.quartz.JobDetail;
import org.quartz.ScheduleBuilder;
import org.quartz.Scheduler;
import org.quartz.SchedulerException;
import org.quartz.Trigger;
import org.quartz.TriggerBuilder;
import org.quartz.impl.StdSchedulerFactory;
public class QuartzMain {
public static void main(String[] args) throws SchedulerException {
JobDetail job = JobBuilder.newJob(QuartzJob.class).build();
Trigger tl = TriggerBuilder.newTrigger().withIdentity("SimpleTrigger").startNow().build();
Scheduler sc = StdSchedulerFactory.getDefaultScheduler();
sc.start();
sc.scheduleJob(job, tl);
}}

To start at a given date:
In the trigger replace .startNow() with .startAt(date) with date being a java.util.Date that represents the date and time it should start.
To connect:
In your Quartz job class call your sendSMS() method (currently your main method in SMSApplication) any variables (such as the phone number) should be passed to the job class in the job builer with .usingJobData("key","value"). See http://www.quartz-scheduler.org/documentation/quartz-2.3.0/ and read through the Tutorials and Cookbook for more information.

kerberos fallback to ntlm java code

I am working on Microsoft Sharepoint integration with SAP. The sharepoint supports 'kerberos fallback to ntlm' method for authentication.
I wish to take advantage of ntlm authentication, as I dont wish to go into kerberos set up. My understanding is that ntlm authentication can be managed with following piece of code however, I am getting '401 Unauthorized' error.
Probably something needs to be added to following line in password authentication.
if (getRequestingScheme().equalsIgnoreCase("negotiate") || getRequestingScheme().equalsIgnoreCase("ntlm") || getRequestingScheme().equalsIgnoreCase("kerberos"))
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.Authenticator;
import java.net.HttpURLConnection;
import java.net.PasswordAuthentication;
import java.net.URL;
import java.lang.Object;
import java.nio.charset.StandardCharsets;
import java.io.OutputStream;
import java.net.InetAddress;
public class Main {
public static void main(String[] argv) throws Exception {
Authenticator.setDefault(new MyAuthenticator());
URL url = new URL ("https://sharepointlink");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoOutput( true );
//conn.setInstanceFollowRedirects( false );
conn.setRequestMethod( "POST" );
conn.setRequestProperty("Accept", "*/*");
//MyAuthenticator m = new MyAuthenticator();
String urlParameters = "tests";
byte[] postData = urlParameters.getBytes( StandardCharsets.UTF_8 );
int postDataLength = postData.length;
byte[] data = ("").getBytes("UTF-8");
OutputStream out = conn.getOutputStream();
out.write(postData);
out.flush();
StringBuilder response = new StringBuilder();
InputStream stream = conn.getInputStream();
// InputStream estream = conn.getErrorStream();
BufferedReader in = new BufferedReader(new InputStreamReader(stream));
String str = "";
while ((str = in.readLine()) != null) {
response.append(str) ;
}
in.close();
System.out.println(response.toString());
// return response.toString() ;
}
}
class MyAuthenticator extends Authenticator {
// static String user = System.getProperty("user");
// static String pass = System.getProperty("pass");
// static String kuser = System.getProperty("kuser");
// static String kpass = System.getProperty("kpass");
// static String showhint = System.getProperty("showhint");
protected PasswordAuthentication getPasswordAuthentication() {
String promptString = getRequestingPrompt();
System.out.println("prompt string " + promptString);
String hostname = getRequestingHost();
System.out.println("host name " + hostname);
InetAddress ipaddr = getRequestingSite();
System.out.println(ipaddr);
int port = getRequestingPort();
RequestorType reqType = getRequestorType();
System.out.println ("reqeust type = " + reqType.toString());
System.out.println ("Protocol type = " + getRequestingProtocol());
System.out.println ("Scheme type = " + getRequestingScheme());
if (getRequestingScheme().equalsIgnoreCase("negotiate") || getRequestingScheme().equalsIgnoreCase("ntlm") || getRequestingScheme().equalsIgnoreCase("kerberos")) {
String krb5user="tp1\\user";
String krb5pass ="pwd";
// get krb5user and krb5pass in your own way
return (new PasswordAuthentication (krb5user, krb5pass.toCharArray()));
}
String username = "TP1\\user";
String password = "pwd";
return new PasswordAuthentication(username, password.toCharArray());
}
}
Can anyone help please.

Jsoup Reddit Image Scraper Over 18 issue

I'm working on an image scraper that scrapes the first page of various subreddits using JSOUP. The issue that arises however is when attempting to scrape a NSFW subreddit, reddit redirects to an over 18 authentication page and the scraper scrapes the authentication page instead. I'm new to scraping and understand this is a noob question, but any help would be much appreciated as I am totally lost.
import java.io.IOException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import java.io.*;
import java.net.URL;
import java.util.logging.Level;
import java.util.logging.Logger;
import java.io.*;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Attributes;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
import java.net.URL;
import java.util.Scanner;
public class javascraper{
public static final String USER_AGENT = "<User-Agent: github.com/dabeermasood:v1.2.3 (by /u/swedenotswiss)>";
public static void main (String[]args) throws MalformedURLException
{
Scanner scan = new Scanner (System.in);
System.out.println("Where do you want to store the files?");
String folderpath = scan.next();
System.out.println("What subreddit do you want to scrape?");
String subreddit = scan.next();
subreddit = ("http://reddit.com/r/" + subreddit);
new File(folderpath + "/" + subreddit).mkdir();
//test
try{
//gets http protocol
Document doc = Jsoup.connect(subreddit).userAgent(USER_AGENT).timeout(0).get();
//get page title
String title = doc.title();
System.out.println("title : " + title);
//get all links
Elements links = doc.select("a[href]");
for(Element link : links){
//get value from href attribute
String checkLink = link.attr("href");
Elements images = doc.select("img[src~=(?i)\\.(png|jpe?g|gif)]");
if (imgCheck(checkLink)){ // checks to see if img link j
System.out.println("link : " + link.attr("href"));
downloadImages(checkLink, folderpath);
}
}
}
catch (IOException e){
e.printStackTrace();
}
}
public static boolean imgCheck(String http){
String png = ".png";
String jpg = ".jpg";
String jpeg = "jpeg"; // no period so checker will only check last four characaters
String gif = ".gif";
int length = http.length();
if (http.contains(png)|| http.contains("gfycat") || http.contains(jpg)|| http.contains(jpeg) || http.contains(gif)){
return true;
}
else{
return false;
}
}
private static void downloadImages(String src, String folderpath) throws IOException{
String folder = null;
//Exctract the name of the image from the src attribute
int indexname = src.lastIndexOf("/");
if (indexname == src.length()) {
src = src.substring(1, indexname);
}
indexname = src.lastIndexOf("/");
String name = src.substring(indexname, src.length());
System.out.println(name);
//Open a URL Stream
URLConnection connection = (new URL(src)).openConnection();
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
} //Delay to comply with rate limiting
connection.setRequestProperty("User-Agent", USER_AGENT);
InputStream in = connection.getInputStream();
OutputStream out = new BufferedOutputStream(new FileOutputStream( folderpath+ name));
for (int b; (b = in.read()) != -1;) {
out.write(b);
}
out.close();
in.close();
}
}

I've posted an answer to authenticate against the server using Jsoup in this link. Basically you need to POST your login ID & password and other required data to the server using:
Connection.Response res = Jsoup.connect(url).data(...).method(Method.Post).execute();, then save the response cookie from the server to keep your session authenticated.

Execution stops automatically at getOutputStream() function Push notification using servlets

I have a problem while getting a push notification on the server side. I am using C2DM sever connection to achieve the push notification. Its working fine on android but not in the Java Servecr. I'm able to get the registration ID and authentication code but the handling stops when it comes to the line:
OutputStream out = conn.getOutputStream();
It does not throw anything, just stops.
If anyone has a solution for this then please guide me. I am adding the whole code so you can go through and tell me where I'm wrong in getting the result.
import org.apache.http.client.methods.HttpPost;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.List;
import javax.net.ssl.HttpsURLConnection;
import javax.print.DocFlavor.INPUT_STREAM;
import org.apache.http.HttpClientConnection;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.DefaultHttpClientConnection;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;
public class Server {
private static String key=null;
// private final static String AUTH = "authentication";
private static final String UPDATE_CLIENT_AUTH = "Update-Client-Auth";
public static final String PARAM_REGISTRATION_ID = "registration_id";
public static final String PARAM_DELAY_WHILE_IDLE = "delay_while_idle";
public static final String PARAM_COLLAPSE_KEY = "collapse_key";
private static final String UTF8 = "UTF-8";
private static String Authcode = null;
// Registration is currently hardcoded
private final static String YOUR_REGISTRATION_STRING = "reg id";
public void getAuthentification() {
System.out.println("check");
//HttpPost post = new HttpPost("http://www.google.com/accounts/ClientLogin");
try {
System.getProperties().put("proxySet", true);
System.getProperties().put("proxyHost","proxy" );
System.getProperties().put("proxyPort","8080");
System.out.println("getAuthentication method called****************************");
URL url=new URL("https://www.google.com/accounts/ClientLogin");
URLConnection connection=url.openConnection();
connection.setDoOutput(true);
HttpURLConnection conn=(HttpURLConnection)connection;
conn.setRequestMethod("POST");
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(true);
StringBuilder content = new StringBuilder();
content.append("Email=").append("xyz#gmail.com");
content.append("&Passwd=").append("asdfgt");
content.append("&service=").append("ac2dm");
content.append("&source=").append("MY_APP-V0.1");
content.append("&accountType=").append("HOSTED_OR_GOOGLE");
OutputStream out = conn.getOutputStream();
out.write(content.toString().getBytes("UTF-8"));
out.close();
int res = conn.getResponseCode();
System.out.println(res + "Success");
StringBuffer resp = new StringBuffer();
if(res == HttpsURLConnection.HTTP_OK){
InputStream in = conn.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(in));
String line = "";
while ((line = rd.readLine()) != null) {
System.out.println(line);
System.out.println("inside");
if (line.startsWith("Auth=")) {
Authcode = line.substring(5);
resp.append(line.substring(5));
System.out.println(line.substring(5));
System.out.println("something to be done here..");
}
}
rd.close();
}
}
}
public void sendMessage() {
try {
System.out.println(YOUR_REGISTRATION_STRING);
System.out.println("Authcode = " + Authcode);
System.getProperties().put("proxySet", true);
System.getProperties().put("proxyHost","proxy" );
System.getProperties().put("proxyPort","8080");
URL url1 = new URL("https://android.apis.google.com/c2dm/send");
System.out.println("here2.5");
HttpURLConnection conn1 = (HttpURLConnection) url1.openConnection();
System.out.println("here2.6");
conn1.setDoInput(true);
conn1.setDoOutput(true);
conn1.setUseCaches(false);
conn1.setRequestMethod("POST");
conn1.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
conn1.setRequestProperty("Authorization", "GoogleLoginauth="+ Authcode);
System.out.println("here2.7");
OutputStream out = conn1.getOutputStream();
System.out.println("send Message method.");
String auth_key="n/a";
if(key!=null) {
auth_key=Authcode;
}
System.out.println("here");
// Send a sync message to this Android device.
StringBuilder postDataBuilder = new StringBuilder();
postDataBuilder.append(PARAM_REGISTRATION_ID)
.append("=").append(YOUR_REGISTRATION_STRING);
System.out.println("here1");
postDataBuilder.append("&").append(PARAM_COLLAPSE_KEY)
.append("=").append("0");
System.out.println("here2");
postDataBuilder.append("&").append("data.payload")
.append("=").append(URLEncoder.encode("Lars war hier", UTF8));
byte[] postData = postDataBuilder.toString().getBytes(UTF8);
conn1.setRequestProperty("Content-Length",
Integer.toString(postData.length));
// Hit the dm URL.
System.out.println("out created");
out.write(postData);
System.out.println("data written");
out.close();
System.out.println("here3");
int responseCode = conn1.getResponseCode();
System.out.println(String.valueOf(responseCode));
// Validate the response code
if (responseCode == 401 || responseCode == 403) {
// The token is too old - return false to retry later, will
// fetch the token
// from DB. This happens if the password is changed or token
// expires. Either admin
// is updating the token, or Update-Client-Auth was received by
// another server,
// and next retry will get the good one from database.
System.out.println("C2DM, Unauthorized - need token");
}
} catch (Exception ignore) {
// the editor that corrected the indentation of the code could not find any code here so he closed all methods to have a syntactically correct class.
}
}
}

Make sure that you are posting message to url from background thread not from the UI thread.
I hope after this change it will work. Good Luck!!!