We Keep Coding

Why is this site returning a IOException with the Java openStream() method?

I am writing a program to output a website's HTML code. I have tested it on some sites such as https://www.stackoverflow.com and it works. However, when I tried running the program with https://www.science.energy.gov, it doesn't work and throws an IOException. If I change the https to http and run it with http://www.science.energy.gov, the program runs but does not print anything. I am not sure why the HTML code for the http website is not displaying.
Below is the relevant code for the HTML extraction program:
import java.net.*;
import java.io.*;
public class URLReader {
public static void main(String[] args) throws Exception {
URL url;
InputStream is = null;
DataInputStream dis;
String line;
try {
url = new URL("https://science.energy.gov/");
is = url.openStream(); // throws an IOException
dis = new DataInputStream(new BufferedInputStream(is));
while ((line = dis.readLine()) != null) {
System.out.println(line);
}
} catch (MalformedURLException mue) {
mue.printStackTrace();
} catch (IOException ioe) {
ioe.printStackTrace();
} finally {
try {
is.close();
} catch (IOException ioe) {
// nothing to see here
}
}
}
}

That's because when you send a request in http for http://science.energy.gov/ it redirects automatically to https, which means the site will reload. And your program is not capable of handling redirect requests. So it just stops. No output no error.
Now about the SSLHandshakeException. The error explains it self, unable to find valid certification path to requested target. Which means your java keystore doesn't have ssl certificate for service you are trying to connect. So you need to obtain the public certificate from the server you're trying to connect to. Read this answer for more information.
Also read,
How to solve javax.net.ssl.SSLHandshakeException Error?
javax.net.ssl.SSLHandshakeException

IOException Java

I tried to run this piece of code without an internet connection, expecting and IOException to trigger:
import java.net.*;
import java.io.*;
public class API_connect {
public static void main(String[] args) {
try {
URL API = new URL("http://api.football-data.org");
URLConnection API_connection = API.openConnection();
}
catch(MalformedURLException exception) {
System.out.print(exception);
}
catch(IOException exception) {
System.out.print(exception);
System.out.print("is something going on here?");
}
}
}
And well... To my surprise nothing was printed, and I can't figure out why. Wouldn't a lack of internet connection be the main reason why an IOException is thrown here?

openConnection() does not actually try to connect:
It should be noted that a URLConnection instance does not establish the actual network connection on creation. This will happen only when calling URLConnection.connect().
Try calling connect() on it.
Alternatively, you could try the following:
new URL(...).openStream().read();
That would actually try to read 1 byte from that url and would fail.

Page replication error in Jsoup

I built one code to download the entire page as HTML with Jsoup. The download part is working as expected. But my problem is - The page is being replicated more than once in the browser when I open the downloaded file and I don't know whats going wrong. Check out the code below:
public class httptest {
static File file;
String crawlingNode;
static BufferedWriter writer = null;
static httptest ht;
public httptest() throws IOException{
file = new File(//***SET HERE YOUR TEST PATH***);
}
private void GetLinks() throws IOException{
Document doc = Jsoup.connect("http://google.com/search?q=mamamia")
.userAgent("Mozilla/5.0 (X11; U; Linux x86_64; en-GB; rv:1.8.1.6) Gecko/20070723 Iceweasel/2.0.0.6 (Debian-2.0.0.6-0etch1)")
.cookie("auth", "token")
.timeout(3000)
.get();
Elements links = doc.select("*");
String crawlingNode = links.html();
System.out.println(crawlingNode);
httptest.WriteOnFile(writer, crawlingNode);
}
private static void OpenWriter(File file){
try {
writer = new BufferedWriter(new FileWriter(file));
} catch (IOException e) {
JOptionPane.showMessageDialog(null, "Failed to open URL Writer");
e.printStackTrace();
}
}
private static void WriteOnFile(BufferedWriter writer, String crawlingNode){
try {
writer.write(crawlingNode);
} catch (IOException e) {
JOptionPane.showMessageDialog(null, "Failed to write URL Node");
e.printStackTrace();
}
}
private static void CloseWriter(BufferedWriter writer){
try {
writer.close();
} catch (IOException e) {
JOptionPane.showMessageDialog(null, "Unable to close URL Writer");
System.err.println(e);
}
}
public static void main (String[] args) throws IOException{
ht = new httptest();
httptest.OpenWriter(file);
ht.GetLinks();
httptest.CloseWriter(writer);
}
}
Some parts of the code might seems weird but remember that this is the SSCCE code version. Any ideas of what might be helpfull please? Thanks in advance.

instead of:
Elements links = doc.select("*");
String crawlingNode = links.html();
System.out.println(crawlingNode);
httptest.WriteOnFile(writer, crawlingNode);
use:
Element links = doc.select("*").first();
String crawlingNode = links.html();
System.out.println(crawlingNode);
httptest.WriteOnFile(writer, crawlingNode);
I think the Elements type is more complex and detailed to use. I found this code change analysing this source: http://jsoup.org/cookbook/extracting-data/attributes-text-html
Anyway, this solution worked out for me.

How to define an explicit constructor for an url object in Java

I am defining a url in processing (or java) with the line:
URL base = new URL("http://www.google.com/");
So I get the following error: Default constructor cannot handle exception type MalformedURLException thrown by implicit super constructor. Must define an explicit constructor. Which I assume is because if my url isn't valid, there's no corresponding catch to catch the error (because I'm declaring my url outside of try).
But, i want to define this url outside of try, because my main try block is in a loop and the url is static. So there's no need to do the definition of the url more than once (and because it's static i'm not afraid of any MalformedURLExceptions).
How do I do that? Is there really no other way than to define the url within a separate try block? (because that seems a bit too much for just a simple static url)
// import libraries
import java.net.*;
import java.io.*;
// url
URL base = new URL("http://www.google.com");
void setup() {
}
void draw() {
try {
// retrieve url
String[] results = loadStrings(base);
// print results
println(results[0]);
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (ConnectException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
// stop looping
noLoop();
}

You just need to define a default constructor for your class which throws MalformedURLException:
class MyClass {
private URL url = new URL("http://www.google.com");
public MyClass() throws MalformedURLException {}
}
The error is happening because the constructor Java is generating for your class isn't able to throw anything, you must specify that it might.
Edit
Seeing your code, your other option is this (I assume setup() gets called before you use the URL)
URL url;
void setup() {
try {
url = new URL("http://www.google.com");
} catch (MalformedURLException ex) {
throw new RuntimeException(ex);
}
}

Since the URL constructor throws a checked exception you have to either catch it or rethrow it. You can create your own factory method that just wraps it in a RuntimeException:
public class URLFactory {
public static URL create(String urlString) {
try {
return new URL(urlString);
} catch (MalformedURLException e) {
throw new RuntimeException(e);
}
}
}
Or maybe just this in your environment?
URL createURL(String urlString) {
try {
return new URL(urlString);
} catch (MalformedURLException e) {
throw new RuntimeException(e);
}
}
And use it instead:
URL base = URLFactory.create("http://www.google.com");

Java URLConnection

Simple stuff, I am learning URLs/Networking in my class and I am trying to display something on a webpage. Later I am going to connect it to a MySQL DB... anyway here is my program:
import java.net.*; import java.io.*;
public class asp {
public static URLConnection
connection;
public static void main(String[] args) {
try {
System.out.println("Hello World!"); // Display the string.
try {
URLConnection connection = new URL("post.php?players").openConnection();
}catch(MalformedURLException rex) {}
InputStream response =
connection.getInputStream();
System.out.println(response);
}catch(IOException ex) {}
} }
It compiles fine... but when I run it I get:
Hello World!
Exception in thread "main" java.lang.NullPointerException
at asp.main(asp.java:17)
Line 17: InputStream response = connection.getInputStream();
Thanks,
Dan

You have a malformed URL, but you wouldn't know because you swallowed its exception!
URL("post.php?players")
This URL is not complete, it misses the host (maybe localhost for you?), and the protocol part, say http so to avoid the malformed URL exception you have to provide the full URL including the protocol
new URL("http://www.somewhere-dan.com/post.php?players")
Use the Sun tutorials on URLConnection first. That snippet is at least known to work, if you substitute the URL in that example with a valid URL you should have a working piece of code.

It's because your URL is not valid. You need to put the full address to the page you are trying to open a connection to. You are catching the malformedurlexception but that means that there is no "connection" object at that point. You have an extra closed bracket after the first catch block it appears as well. You should put the line that you are getting the null pointer for and the system.out.println above the catch blocks
import java.net.*; import java.io.*;
public class asp {
public static URLConnection connection;
public static void main(String[] args) {
try {
System.out.println("Hello World!"); // Display the string.
try {
URLConnection connection = new URL("http://localhost/post.php?players").openConnection();
InputStream response = connection.getInputStream();
System.out.println(response);
}catch(MalformedURLException rex) {
System.out.println("Oops my url isn't right");
}catch(IOException ex) {}
}
}