We Keep Coding

Why is this site returning a IOException with the Java openStream() method?

I am writing a program to output a website's HTML code. I have tested it on some sites such as https://www.stackoverflow.com and it works. However, when I tried running the program with https://www.science.energy.gov, it doesn't work and throws an IOException. If I change the https to http and run it with http://www.science.energy.gov, the program runs but does not print anything. I am not sure why the HTML code for the http website is not displaying.
Below is the relevant code for the HTML extraction program:
import java.net.*;
import java.io.*;
public class URLReader {
public static void main(String[] args) throws Exception {
URL url;
InputStream is = null;
DataInputStream dis;
String line;
try {
url = new URL("https://science.energy.gov/");
is = url.openStream(); // throws an IOException
dis = new DataInputStream(new BufferedInputStream(is));
while ((line = dis.readLine()) != null) {
System.out.println(line);
}
} catch (MalformedURLException mue) {
mue.printStackTrace();
} catch (IOException ioe) {
ioe.printStackTrace();
} finally {
try {
is.close();
} catch (IOException ioe) {
// nothing to see here
}
}
}
}

That's because when you send a request in http for http://science.energy.gov/ it redirects automatically to https, which means the site will reload. And your program is not capable of handling redirect requests. So it just stops. No output no error.
Now about the SSLHandshakeException. The error explains it self, unable to find valid certification path to requested target. Which means your java keystore doesn't have ssl certificate for service you are trying to connect. So you need to obtain the public certificate from the server you're trying to connect to. Read this answer for more information.
Also read,
How to solve javax.net.ssl.SSLHandshakeException Error?
javax.net.ssl.SSLHandshakeException

IOException Java

I tried to run this piece of code without an internet connection, expecting and IOException to trigger:
import java.net.*;
import java.io.*;
public class API_connect {
public static void main(String[] args) {
try {
URL API = new URL("http://api.football-data.org");
URLConnection API_connection = API.openConnection();
}
catch(MalformedURLException exception) {
System.out.print(exception);
}
catch(IOException exception) {
System.out.print(exception);
System.out.print("is something going on here?");
}
}
}
And well... To my surprise nothing was printed, and I can't figure out why. Wouldn't a lack of internet connection be the main reason why an IOException is thrown here?

openConnection() does not actually try to connect:
It should be noted that a URLConnection instance does not establish the actual network connection on creation. This will happen only when calling URLConnection.connect().
Try calling connect() on it.
Alternatively, you could try the following:
new URL(...).openStream().read();
That would actually try to read 1 byte from that url and would fail.

Java: GarbageCollectorMXBean getCollectionCount throws java.io.IOException: The client has been closed

I am writing a Java application where I am using Java GarbageCollectorMXBean APIs to get the collection count at regular intervals (for every 5 seconds). Below is the program I have written to do the task.
import java.io.IOException;
import java.lang.management.GarbageCollectorMXBean;
import java.lang.management.ManagementFactory;
import java.util.HashMap;
import java.util.Map;
import javax.management.MBeanServerConnection;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
public class JMXTest {
public static final String GC_NAME = "java.lang:name=MarkSweepCompact,type=GarbageCollector";
private static GarbageCollectorMXBean garbageCollectorMXBean;
private static JMXConnector jmxConnector;
private static MBeanServerConnection mbsc;
public static void main(String[] args) throws Exception {
String rmiHostname = "jmxserver";
String defaultUrl = "service:jmx:rmi:///jndi/rmi://" + rmiHostname + ":1999/jmxrmi";
JMXServiceURL jmxServiceURL = new JMXServiceURL(defaultUrl);
Map<String,Object> jmxCredentials = new HashMap<String,Object>();
String[] credentials = new String[]{"jmxusername", "jmxpassword"};
jmxCredentials.put("jmx.remote.credentials", credentials);
boolean run = true;
while(run){
try {
if(garbageCollectorMXBean == null){
if (mbsc == null){
jmxConnector = JMXConnectorFactory.connect(jmxServiceURL, jmxCredentials);
mbsc = jmxConnector.getMBeanServerConnection();
}
garbageCollectorMXBean = ManagementFactory.newPlatformMXBeanProxy(mbsc, GC_NAME,GarbageCollectorMXBean.class);
}
long count = garbageCollectorMXBean.getCollectionCount();
System.out.println("Garbage Collector count = " + count);
} catch (Exception e) {
e.printStackTrace();
garbageCollectorMXBean = null;
if (jmxConnector != null)
{
try
{
jmxConnector.close();
} catch (IOException ioe) {}
jmxConnector = null;
}
mbsc = null;
}
Thread.currentThread().sleep(5000);
}
}
}
The program runs fine, but sometimes it starts giving following IOException repeatedly in every loop.
Exception in thread "main" java.io.IOException: The client has been closed.
at java.util.TimerThread.run(Timer.java:505)
at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:94)
at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.gotIOException(ClientCommunicatorAdmin.java:54)
at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1470)
at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:906)
at com.ibm.lang.management.OpenTypeMappingIHandler$6.run(OpenTypeMappingIHandler.java:506)
at java.security.AccessController.doPrivileged(AccessController.java:330)
at com.ibm.lang.management.OpenTypeMappingIHandler.invokeAttributeGetter(OpenTypeMappingIHandler.java:501)
at com.ibm.lang.management.OpenTypeMappingIHandler.invoke(OpenTypeMappingIHandler.java:121)
at com.sun.proxy.$Proxy112.getCollectionCount(Unknown Source)
at JMXTest.main(JMXTest.java:48)
Looking at the code, any exception will get caught in catch block where all fields will be initialized to null and in the next loop all the fields will be reinitialized. But, looking at the logs, once the exception starts coming, I get above exception only at getCollectionCount() call in every loop. I wonder even though the objects are re-initialized, every time I get same exception.
I am looking at following things from the above information
In what all cases, we get this exception java.io.IOException: The client has been closed. in the above scenario. I know, if we call jmxConnector.close() and then use the already created garbageCollectorMXBean object to get the collection count, we get this. But my code does not follow that path.
For the above issue, does jmxserver remote JMX server contributes? I tried to reproduce by stopping/restarting the remote JMX server, but could not do it.

judging from the stack trace last line :
at com.sun.proxy.$Proxy112.getCollectionCount(Unknown Source)
at JMXTest.main(JMXTest.java:48)
it seems the problem is with line of code
if (jmxConnector != null)
{
try
{
jmxConnector.close();
} catch (IOException ioe) {}
jmxConnector = null; //...line no.48
}
mbsc = null; //......this is probably causing the issue
#raghavendra , as you acquire the "MBeanServerConnection" object from the "JMXConnector" , you should close the objects / nullify them in that order i.e. change your code to
if (jmxConnector != null)
{
try
{
mbsc = null; //...object handle assigned null before closing the connector
jmxConnector.close();
} catch (IOException ioe) {}
jmxConnector = null;
}

Connections to SSL ports using NIO stay established after disconnect

In our application we need to check if certain ports of certain host are available for communication. At stage of this check we do not proceed with real communication - we need just to check if ports are open. As many ports has to be checked at once we originally used NIO approach (Selector + SocketChannel classes):
package test;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.SocketChannel;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
public class TestNIO {
public static void main(final String... params) {
final List<String> portsToCheck = Arrays.asList(new String[] {"443", "5989"});
final List<String> openPorts = new ArrayList<String>();
final String host = "<SOME_IP>";
final int timeout = 5000;
Selector selector = null;
if (!portsToCheck.isEmpty()) {
try {
selector = Selector.open();
for (final String port : portsToCheck) {
final SocketChannel channel = SocketChannel.open();
channel.configureBlocking(false);
channel.connect(new InetSocketAddress(host, Integer.valueOf(port)));
channel.register(selector, SelectionKey.OP_CONNECT);
}
final int readyChannels = selector.select(timeout);
if (readyChannels != 0) {
final Iterator<SelectionKey> it = selector.selectedKeys().iterator();
while (it.hasNext()) {
final SelectionKey selKey = it.next();
try {
if (selKey.isValid() && selKey.isConnectable()) {
final SocketChannel channel = (SocketChannel) selKey.channel();
try {
if (channel.finishConnect()) {
openPorts.add(String.valueOf(channel.socket().getPort()));
}
} catch (final IOException ex) {
ex.printStackTrace();
} finally {
channel.close();
}
}
} catch (final Exception ex) {
ex.printStackTrace();
} finally {
selKey.cancel();
}
it.remove();
}
}
} catch (final IOException ex) {
ex.printStackTrace();
} finally {
try {
if (selector != null && selector.isOpen()) {
selector.close();
}
} catch (IOException ex) {
ex.printStackTrace();
}
System.out.print("Open ports: " + openPorts.toString());
}
}
}
}
This approach worked successfully for years at let's say hundreds of customers till at one of our customer this approach leaded to a problem. Namely, connections from client (where this check is running, it's Windows Server 2012 R2) to server (it's ESXi) at just one of SSL-ports stay established and never closed till restart of the server. This happens just with one of SSL-ports (standard 443), for example with another SSL-port - 5989 (it's HTTPS CIM server) this does not happen. Looks like this is because of some configuration on Windows side: 1. Happens just with one of several HTTPS ports; 2. Happens with any ESXi servers connected to this Windows client; 3. Does not happen with another Windows client connected to the same ESXi servers. Problem is that customer does not wish very much to cooperate with us finding the root cause and we have to guess it ourselves. We used another classic approach to check SSL-ports which works fine even in this problematic system. Here it is:
package test;
import java.io.IOException;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import java.net.URL;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSession;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
public class TestHttpUrlConnection {
public static void main(final String... params) {
final List<String> portsToCheck = Arrays.asList(new String[] {"443", "5989"});
final List<String> openPorts = new ArrayList<String>();
final String host = "<SOME_IP>";
final int timeout = 5000;
if (!portsToCheck.isEmpty()) {
trustAllHttpsCertificates();
HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() {
public boolean verify(String urlHostName, SSLSession session) {
return true;
}
});
for (final String port : portsToCheck) {
HttpsURLConnection connection = null;
OutputStreamWriter out = null;
try {
connection = (HttpsURLConnection) new URL(
"https://" + host + ":" + Integer.valueOf(port)).openConnection();
connection.setDoOutput(true);
connection.setConnectTimeout(timeout);
final OutputStream os = connection.getOutputStream();
out = new OutputStreamWriter(os, "UTF8");
out.close();
openPorts.add(port);
} catch(final IOException ex) {
ex.printStackTrace();
} finally {
if (out != null) {
try {
out.close();
} catch (final IOException ex) {
ex.printStackTrace();
}
}
if (connection != null) {
connection.disconnect();
}
}
}
System.out.print("Open ports: " + openPorts.toString());
}
}
private static void trustAllHttpsCertificates() {
try {
final TrustManager[] trustAllCerts = new TrustManager[1];
trustAllCerts[0] = new TrustAllManager();
final SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, null);
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
} catch (final Exception ex) {
ex.printStackTrace();
}
}
private static class TrustAllManager implements X509TrustManager {
public X509Certificate[] getAcceptedIssuers() {
return null;
}
public void checkServerTrusted(final X509Certificate[] certs, final String authType) throws CertificateException {
// Empty
}
public void checkClientTrusted(X509Certificate[] certs, String authType) throws CertificateException {
// Empty
}
}
}
But customer wants us to tell him the reason why one approach works and another does not. Can anyone help?
UPDATE
I've found out that on that problematic system even following code leads to a situation that each connection stays ESTABLISHED and not released back to system. This is not NIO and explicit close() on socket invoked:
Socket sock = new Socket();
SocketAddress serverSocketAddress = new InetSocketAddress(host, port);
try {
sock.connect(serverSocketAddress, timeout);
if (sock.isConnected()) {
openPorts.add(port);
}
} catch (IOException e) {
ex.printStackTrace();
} finally {
if (sock != null) {
try {
sock.close();
} catch (IOException e) {
ex.printStackTrace();
}
}
}
Setting keepAlive to false does not change the situation.
UPDATE 2
Problem repeated also on nonSSL-port (135, it's Hyper-V virtualization). And what confuses me most is that after restart of guest OS with which connections were established and and after stopping software which opened these connections, they are still marked as established on client machine. I consider there is really something wrong with the system itself (and has nothing to do with our Java code), but what exactly is wrong...
UPDATE 3
The problem was caused by TrendMicro's Anti-Virus software "Virus Buster". It prevented connections to be closed normally.

Registered channels aren't completely closed until the next select() call. This is documented somewhere which I can never find when I look for it.
NB your trustAllCertificates() method does nothing useful except implement a radical insecurity, and calling it once per open socket that shouldn't be open at all appears to be completely pointless.

The problem was caused by TrendMicro's Anti-Virus software "Virus Buster". It prevented connections to be closed normally.

Help me to fix a bug in this thread example

I am trying to create a thread to simply send the text to client. However, if you copy this code to IDE, you will see that there is a red underline under client.getOutputStream(). I do not know what is wrong here. The IDE says "Unhandled exception type IOException". Could anybody tell me?
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.net.Socket;
import java.util.ArrayList;
import java.util.List;
public class ServerStudentThread extends Thread {
Socket client;
public ServerStudentThread(Socket x) {
client = x;
}
public void run() {
// create object to send information to client
PrintWriter out = new PrintWriter(client.getOutputStream(),true);
out.println("Student name: ");//send text to client;
}
}
For reference, here is the code that calls the thread.
import java.io.*;
import java.net.*;
import java.util.*;
public class Server2 {
public static void main(String args[]) throws Exception {
int PORT = 5555; // Open port 5555
//open socket to listen
ServerSocket server = new ServerSocket(PORT);
Socket client = null;
while (true) {
System.out.println("Waiting for client...");
// open client socket to accept connection
client = server.accept();
System.out.println(client.getInetAddress()+" contacted ");
System.out.println("Creating thread to serve request");
ServerStudentThread student = new ServerStudentThread(client);
student.start();
}
}
}

It's probably that getOutputStream() can throw an exception and you're not catching it, try putting a try / catch (IOException e) around the block of code.
public void run() {
try {
// create object to send information to client
PrintWriter out = new PrintWriter(client.getOutputStream(),true);
out.println("Student name: ");//send text to client;
} catch (IOException e) {
throw new RuntimeException("It all went horribly wrong!", e);
}
}

So you need to add a try/catch block to handle the I/O exception.
Read the section on Exceptions from the Java tutorial.

From the javadoc:
public OutputStream getOutputStream() throws IOException
IOException is a checked exception. You need to use a try/catch block to handle that possibility.

Kalla,
You need to either put the line in between try/catch block or declare run to throw IOException