We Keep Coding

Is it possible to retrieve a file from a remote path using java?

I've been tasked with a project to automate a process in which we extract a file from a WinSCP client daily. So far I've been able to automate the login and setup a recurring schedule for the code to run; however it seems I've hit a bump in the road. When I attempt to locate a file for retrieval nothing happens. This is because the file I wish to access is through a remote directory. I'm almost positive that the code I've written is bug free. I am just unsure if specify a certain path which java can locate the file. I have no idea how to tell the java code where to extract this file from. Any thoughts?

You can try and use the code below:
More details can be found here.
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import org.apache.commons.net.ftp.FTP;
import org.apache.commons.net.ftp.FTPClient;
/**
* A program demonstrates how to upload files from local computer to a remote
* FTP server using Apache Commons Net API.
* #author www.codejava.net
*/
public class FTPDownloadFileDemo {
public static void main(String[] args) {
String server = "www.myserver.com";
int port = 21;
String user = "user";
String pass = "pass";
FTPClient ftpClient = new FTPClient();
try {
ftpClient.connect(server, port);
ftpClient.login(user, pass);
ftpClient.enterLocalPassiveMode();
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
// APPROACH #1: using retrieveFile(String, OutputStream)
String remoteFile1 = "/test/video.mp4";
File downloadFile1 = new File("D:/Downloads/video.mp4");
OutputStream outputStream1 = new BufferedOutputStream(new FileOutputStream(downloadFile1));
boolean success = ftpClient.retrieveFile(remoteFile1, outputStream1);
outputStream1.close();
if (success) {
System.out.println("File #1 has been downloaded successfully.");
}
// APPROACH #2: using InputStream retrieveFileStream(String)
String remoteFile2 = "/test/song.mp3";
File downloadFile2 = new File("D:/Downloads/song.mp3");
OutputStream outputStream2 = new BufferedOutputStream(new FileOutputStream(downloadFile2));
InputStream inputStream = ftpClient.retrieveFileStream(remoteFile2);
byte[] bytesArray = new byte[4096];
int bytesRead = -1;
while ((bytesRead = inputStream.read(bytesArray)) != -1) {
outputStream2.write(bytesArray, 0, bytesRead);
}
success = ftpClient.completePendingCommand();
if (success) {
System.out.println("File #2 has been downloaded successfully.");
}
outputStream2.close();
inputStream.close();
} catch (IOException ex) {
System.out.println("Error: " + ex.getMessage());
ex.printStackTrace();
} finally {
try {
if (ftpClient.isConnected()) {
ftpClient.logout();
ftpClient.disconnect();
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}

Reading URL, getting read times out error

Hi I am using following code to reading URL:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
public class JavaHttpUrlConnectionReader
{
public static void main(String[] args)
throws Exception
{
new JavaHttpUrlConnectionReader();
}
public JavaHttpUrlConnectionReader()
{
try
{
String myUrl = "http://epaperbeta.timesofindia.com/NasData/PUBLICATIONS/THETIMESOFINDIA/Delhi/2015/06/09/PageIndex/09_06_2015.xml";
// if your url can contain weird characters you will want to
// encode it here, something like this:
// myUrl = URLEncoder.encode(myUrl, "UTF-8");
String results = doHttpUrlConnectionAction(myUrl);
System.out.println(results);
}
catch (Exception e)
{
// deal with the exception in your "controller"
}
}
/**
* Returns the output from the given URL.
*/
private String doHttpUrlConnectionAction(String desiredUrl)
throws Exception
{
URL url = null;
BufferedReader reader = null;
StringBuilder stringBuilder;
try
{
// create the HttpURLConnection
url = new URL(desiredUrl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
// just want to do an HTTP GET here
connection.setRequestMethod("GET");
// uncomment this if you want to write output to this url
//connection.setDoOutput(true);
// give it 15 seconds to respond
connection.setReadTimeout(35*1000);
connection.connect();
// read the output from the server
reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
stringBuilder = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null)
{
stringBuilder.append(line + "\n");
}
return stringBuilder.toString();
}
catch (Exception e)
{
e.printStackTrace();
throw e;
}
finally
{
// close the reader; this can throw an exception too, so
// wrap it in another try/catch block.
if (reader != null)
{
try
{
reader.close();
}
catch (IOException ioe)
{
ioe.printStackTrace();
}
}
}
}
}
It gives me following error:
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1072)
at JavaHttpUrlConnectionReader.doHttpUrlConnectionAction(JavaHttpUrlConnectionReader.java:77)
at JavaHttpUrlConnectionReader.<init>(JavaHttpUrlConnectionReader.java:33)
at JavaHttpUrlConnectionReader.main(JavaHttpUrlConnectionReader.java:21)
Kindly tell me the reason why it occurs, and solution for it.
When I run this code outside of my office LAN, it is working fine. but not in office LAN.
Thanks & Regards
Abhishek

Your URL:
http://epaperbeta.timesofindia.com/NasData/PUBLICATIONS/THETIMESOFINDIA/Delhi/2015/06/09/PageIndex/09_06_2015.xml
is not accessible without a proxy (for example I can't access it from here), no wonder why cannot be read from the stream.
Check your proxy settings. you could try the url in the browser with/without proxy and see the difference.
As #Jens commented look at this.

Java: ConnectException/Cannot find or load Main-Class

So in a nutshell, I'm just trying to get a small working skeleton program that I can use to sort of learn about Http communication and "feel" my way around to figure out what I will eventually need for a bigger program I am working on. This particular code here is actually just a chopped up version of an example from the Apache libraries. I could compile the examples listed on the Apache website, but they didn't run properly, giving a "java.net.ConnectException". I figured it had to do with Windows c-blocking a program like this from making a connection, and that I would need to run it as an administrator. I then tried taking the code and throwing it into an executable jar file, but I get a Cannot-find-or-load-main-class error. Am I an idiot or is the Apache library a little outdated/not fit for Win 8/something else?
Code below:
package NewProject;
import java.net.Socket;
import org.apache.http.ConnectionReuseStrategy;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.impl.DefaultBHttpClientConnection;
import org.apache.http.impl.DefaultConnectionReuseStrategy;
import org.apache.http.message.BasicHttpRequest;
import org.apache.http.protocol.HttpCoreContext;
import org.apache.http.protocol.HttpProcessor;
import org.apache.http.protocol.HttpProcessorBuilder;
import org.apache.http.protocol.HttpRequestExecutor;
import org.apache.http.protocol.RequestConnControl;
import org.apache.http.protocol.RequestContent;
import org.apache.http.protocol.RequestExpectContinue;
import org.apache.http.protocol.RequestTargetHost;
import org.apache.http.protocol.RequestUserAgent;
import org.apache.http.util.EntityUtils;
class NewProject
{
public static void main(String[] args) throws Exception
{
HttpProcessor httpproc = HttpProcessorBuilder.create()
.add(new RequestContent())
.add(new RequestTargetHost())
.add(new RequestConnControl())
.add(new RequestUserAgent("Test/1.1"))
.add(new RequestExpectContinue(true)).build();
HttpRequestExecutor httpexecutor = new HttpRequestExecutor();
HttpCoreContext coreContext = HttpCoreContext.create();
HttpHost host = new HttpHost("localhost", 8080);
coreContext.setTargetHost(host);
Out os = new Out("TestOut.txt");
DefaultBHttpClientConnection conn = new DefaultBHttpClientConnection(8 * 1024);
ConnectionReuseStrategy connStrategy = DefaultConnectionReuseStrategy.INSTANCE;
try
{
String[] targets =
{
"http://www.google.com/"
};
for (int i = 0; i < targets.length; i++)
{
if (!conn.isOpen())
{
Socket socket = new Socket(host.getHostName(), host.getPort());
conn.bind(socket);
}
BasicHttpRequest request = new BasicHttpRequest("GET", targets[i]);
os.println(">> Request URI: " + request.getRequestLine().getUri());
httpexecutor.preProcess(request, httpproc, coreContext);
HttpResponse response = httpexecutor.execute(request, conn, coreContext);
httpexecutor.postProcess(response, httpproc, coreContext);
os.println("<< Response: " + response.getStatusLine());
os.println(EntityUtils.toString(response.getEntity()));
os.println("==============");
if (!connStrategy.keepAlive(response, coreContext))
{
conn.close();
}
else
{
os.println("Connection kept alive...");
}
}
}
catch (IndexOutOfBoundsException iob)
{
os.println("What happened here?");
}
finally
{
conn.close();
}
return;
}
}

... they didn't run properly, giving a "java.net.ConnectException"
That could be caused by lots of things. There are clues in the exception message ... which you chose not to share with us.
... "Cannot find or load Main-Class"
Again multiple possible causes, and there are clues in the exception message ... which you chose not to share with us.
But the fact that you have created a JAR file plus the "Main-Class" hint in the error message fragment you provided suggest that you made a mistake in the creation of the JAR file; i.e. you used the wrong name for the "Main-Class" attribute.
Given that source code, the "Main-Class" attribute should be "NewProject.NewProject". I suspect you set it to something else.
A second possibility is that you haven't handled the dependency on the Apache library correctly. The Apache classes need to be on the classpath specified by the JAR file. (You can't use a -cp argument or $CLASSPATH when you launch with java -jar.)
Am I an idiot or is the Apache library a little outdated/not fit for Win 8/something else?
There is nothing wrong with the Apache library.

The code you posted seems a little low level (e.g. interacting directly with Socket connections). The code posted below should give you what it sounds like you are looking for. The classes used also give you a lot of inroads into setting and getting http parameters (e.g. headers, time-outs, etc).
package org.yaorma.example.http.client;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;
public class HttpClientExample {
public static void main(String[] args) throws Exception {
String response;
response = get("http://www.google.com");
System.out.println("RESPONSE FROM GET -----------------------------------------");
System.out.println(response);
response = post("http://httpbin.org/post", "This is the message I posted to httpbin.org/post");
System.out.println("RESPONSE FROM POST -----------------------------------------");
System.out.println(response);
}
/**
* Method to post a request to a given URL.
*/
public static String post(String urlString, String message) {
try {
// get a connection
URL url = new URL(urlString);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
// set the parameters
conn.setRequestMethod("POST");
conn.setDoOutput(true);
// send the message
OutputStream os = conn.getOutputStream();
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(os, "UTF-8"));
writer.write(message);
writer.flush();
writer.close();
os.close();
// get the response
conn.connect();
InputStream content = (InputStream) conn.getInputStream();
// read the response
BufferedReader in = new BufferedReader(new InputStreamReader(content));
String rtn = "";
String line;
while ((line = in.readLine()) != null) {
rtn += line + "\n";
}
return rtn;
} catch (Exception exp) {
throw new RuntimeException(exp);
}
}
/**
* Method to do a get from a given URL.
*/
public static String get(String urlString) {
try {
// get a connection
URL url = new URL(urlString);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
// set the parameters
conn.setRequestMethod("GET");
conn.setDoOutput(true);
// get the response
conn.connect();
InputStream content = (InputStream) conn.getInputStream();
// read the response
BufferedReader in = new BufferedReader(new InputStreamReader(content));
String rtn = "";
String line;
while ((line = in.readLine()) != null) {
rtn += line + "\n";
}
return rtn;
} catch (Exception exp) {
throw new RuntimeException(exp);
}
}
}

How to fix error 502 status

I am using Jsoup Java HTML parser to fetch images from a particular URL. But some of the images are throwing a status 502 error code and are not saved to my machine. Here is the code snapshot i have used:-
String url = "http://www.jabong.com";
String html = Jsoup.connect(url.toString()).get().html();
Document doc = Jsoup.parse(html, url);
images = doc.select("img");
for (Element element : images) {
String imgSrc = element.attr("abs:src");
log.info(imgSrc);
if (imgSrc != "") {
saveFromUrl(imgSrc, dirPath+"/" + nameCounter + ".jpg");
try {
Thread.sleep(3000);
} catch (InterruptedException e) {
log.error("error in sleeping");
}
nameCounter++;
}
}
And the saveFromURL function looks like this:-
public static void saveFromUrl(String Url, String destinationFile) {
try {
URL url = new URL(Url);
InputStream is = url.openStream();
OutputStream os = new FileOutputStream(destinationFile);
byte[] b = new byte[2048];
int length;
while ((length = is.read(b)) != -1) {
os.write(b, 0, length);
}
is.close();
os.close();
} catch (IOException e) {
log.error("Error in saving file from url:" + Url);
//e.printStackTrace();
}
}
I searched on internet about status code 502 but it says error is due to bad gateway. I don't understand this. One of the possible things i am thinking that this error may be because of I am sending get request to images in loop. May be webserver is not able handle to this much load so denying the request to the images when previous image is not sent.So I tried to put sleep after fetching every image but no luck :(
Some advices please

Here's a full code example that works for me...
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.Authenticator;
import java.net.HttpURLConnection;
import java.net.InetSocketAddress;
import java.net.MalformedURLException;
import java.net.Proxy;
import java.net.SocketAddress;
import java.net.URL;
public class DownloadImage {
public static void main(String[] args) {
// URLs for Images we wish to download
String[] urls = {
"http://cdn.sstatic.net/stackoverflow/img/apple-touch-icon.png",
"http://www.google.co.uk/images/srpr/logo3w.png",
"http://i.microsoft.com/global/en-us/homepage/PublishingImages/sprites/microsoft_gray.png"
};
for(int i = 0; i < urls.length; i++) {
downloadFromUrl(urls[i]);
}
}
/*
Extract the file name from the URL
*/
private static String getOutputFileName(URL url) {
String[] urlParts = url.getPath().split("/");
return "c:/temp/" + urlParts[urlParts.length-1];
}
/*
Assumes there is no Proxy server involved.
*/
private static void downloadFromUrl(String urlString) {
InputStream is = null;
FileOutputStream fos = null;
try {
URL url = new URL(urlString);
System.out.println("Reading..." + url);
HttpURLConnection conn = (HttpURLConnection)url.openConnection(proxy);
is = conn.getInputStream();
String filename = getOutputFileName(url);
fos = new FileOutputStream(filename);
byte[] readData = new byte[1024];
int i = is.read(readData);
while(i != -1) {
fos.write(readData, 0, i);
i = is.read(readData);
}
System.out.println("Created file: " + filename);
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
finally {
if(is != null) {
try {
is.close();
} catch (IOException e) {
System.out.println("Big problems if InputStream cannot be closed");
}
}
if(fos != null) {
try {
fos.close();
} catch (IOException e) {
System.out.println("Big problems if FileOutputSream cannot be closed");
}
}
}
System.out.println("Completed");
}
}
You should see the following ouput on your console...
Reading...http://cdn.sstatic.net/stackoverflow/img/apple-touch-icon.png
Created file: c:/temp/apple-touch-icon.png
Completed
Reading...http://www.google.co.uk/images/srpr/logo3w.png
Created file: c:/temp/logo3w.png
Completed
Reading...http://i.microsoft.com/global/en-us/homepage/PublishingImages/sprites/microsoft_gray.png
Created file: c:/temp/microsoft_gray.png
Completed
So that's a working example without a Proxy server involved.
Only if you require authentication with a proxy server here's an additional Class that you'll need based on this Oracle technote
import java.net.Authenticator;
import java.net.PasswordAuthentication;
public class ProxyAuthenticator extends Authenticator {
private String userName, password;
public ProxyAuthenticator(String userName, String password) {
this.userName = userName;
this.password = password;
}
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(userName, password.toCharArray());
}
}
And to use this new Class you would use the following code in place of the call to openConnection() shown above
...
try {
URL url = new URL(urlString);
System.out.println("Reading..." + url);
Authenticator.setDefault(new ProxyAuthenticator("username", "password");
SocketAddress addr = new InetSocketAddress("proxy.server.com", 80);
Proxy proxy = new Proxy(Proxy.Type.HTTP, addr);
HttpURLConnection conn = (HttpURLConnection)url.openConnection(proxy);
...

Your problem sounds like HTTP communication issues, so you are probably better off trying to use a library to handle the communication side of things. Take a look at Apache Commons HttpClient.
Some notes about your code example. You haven't used a URLConnection object so it's not clear what the behaviour will be in regards to the Web/Proxy servers and closing resources cleanly, etc. The HttpCommon library mentioned will help in this aspect.
There also seems to be some examples of doing what you want using J2ME libararies. Not something I have used personally but may also help you out.

Java, HttpUrlConnection. Problem with getResponseCode()

I try to create the simplest Simplest WebServer and Client using HTTP. (Please, don't tell me to using Apache HTTPClient).
Client: try to PUT some file to Server.
// **PUT**
if(REQUEST.toUpperCase().equals("PUT")) {
File sourceFile = new File(fileName);
if(!sourceFile.canRead()) {
System.out.println("Have not access to this file...");
return;
}
try {
BufferedInputStream is = new BufferedInputStream(new FileInputStream(sourceFile));
URL url = new URL("http://" + HOST+":"+PORT);
System.setProperty("http.keepAlive", "true");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setRequestMethod("PUT");
connection.setRequestProperty("Content-Type", "Application/octet-stream");
connection.setRequestProperty("Content-Length",Long.toString(sourceFile.length()));
connection.addRequestProperty("Content-disposition","attachment; filename="+fileName);
BufferedOutputStream os = new BufferedOutputStream(connection.getOutputStream());
byte[] buf = new byte[sizeArr];
int r = 1;
while((r = is.read(buf)) > 0) {
os.write(buf, 0, r);
}
os.flush();
os.close();
System.out.println("Waiting for the response...");//this is written to console
System.out.println(connection.getResponseCode());//HERE infinite waiting
is.close();
} catch (MalformedURLException ex) {
ex.printStackTrace();
} catch (IOException ex) {
ex.printStackTrace();
}
}
On Server: if Request == PUT, then:
// **PUT**
if (header.toUpperCase().equals("PUT")) {
System.setProperty("http.keepAlive", "true");
String fileName = null;
if((fileName = extract(request.toUpperCase(),"FILENAME=","\n")) == null) {
fileName = "UnknownFile.out";
}
try {
File sourceFile = new File(fileName);
BufferedOutputStream osFile = new BufferedOutputStream
(new FileOutputStream(sourceFile));
byte[] locbuf = new byte[sizeArr];
int locr = 1;
while((locr = is.read(locbuf)) > 0) {
System.out.println("locr= "+locr);//this is written to console
osFile.write(locbuf, 0, locr);
}
System.out.println("Ending to record the data to the file.");
// this is NOT written to console
osFile.flush();
osFile.close();
}
catch(IOException ex) {
os.write(CodeRequest("500 Internal Server Error").getBytes());
os.close();
ex.printStackTrace();
return;
}
System.out.println("Trying to send 200 OK");
os.write(CodeRequest("200 OK").getBytes());
os.flush();
os.close(); // where os = clientSocket.getOutputStream();
}
Why doesn't the Client get a Response from the Server? If I interrupted the Client's infinite loop, then WebServer would correctly record data to file. But Client will never know that his file was normally uploaded to the server. If I comment out this statement on Client:
// System.out.println(connection.getResponseCode());
Then Client correctly exit from loop and ends. But Server doesn't even write to console this:
while((locr = is.read(locbuf)) > 0) {
System.out.println("locr= "+locr);//this is NOT written to console
osFile.write(locbuf, 0, locr);
}
Server ONLY writes this to console this:
localExcString index out of range: -1
without any Error message.
What's wrong?

Your example code for the server doesn't show the declaration and initialisation of 'is'.
However, my guess is that since the session is keep alive the call to is.read() will block until some data arrives. You have set the content length in the client, so I would be expecting to see the read loop complete when that amount of data has been successfully read.

What string is the CodeRequest method returning in the server code? I think that the problem may be that you are not putting a CRLF at the end of the status line and another one at the end of the response header. For details, read the HTTP 1.1 specification.

You need to call URLConnection#getInputStream() after the write to actually send the request (and thus retrieve the response). Only after this point you can request the response status. This is implemented so because it might take longer to build the request body than to actually sending it and also because there's actually no point of having a request if you're not interested in the response.
Edit sorry, I was wrong. The getResponseCode() already implicitly calls getInputStream().
Anyway, I created a small testcase (see SSCCE for more info) and it just works fine. Try it and see if it works in your case as well.
Client:
import java.io.IOException;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
public class Test {
public static void main(String[] args) throws Exception {
String data = "Hello!";
URL url = new URL("http://localhost:8080/playground/test");
OutputStream output = null;
try {
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("PUT");
output = connection.getOutputStream();
output.write(data.getBytes());
System.out.println("Response code: " + connection.getResponseCode());
} finally {
if (output != null) try { output.close(); } catch (IOException ignore) {}
}
}
}
Server:
package mypackage;
import java.io.IOException;
import java.io.InputStream;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class Test extends HttpServlet {
protected void doPut(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
{
InputStream input = null;
try {
input = request.getInputStream();
int data;
while ((data = input.read()) > -1) {
System.out.write(data);
}
System.out.println();
} finally {
if (input != null) try { input.close(); } catch (IOException ignore) {}
}
}
}
Result should be a Response code: 200 in the client's stdout and a Hello! in the server's stdout.
Hope this helps in pinpointing the cause of your problem.

On Server:
String strFileLen = extract(request.toUpperCase().trim(),"CONTENT-LENGTH:","\n");
long fileLength = 0;
if(strFileLen != null) {
fileLength = Long.parseLong(strFileLen.trim());
System.out.println("fileLength= "+fileLength);
};
byte[] locbuf = new byte[sizeArr];
int locr = 1;long sumLen = 0;
if(fileLength != 0) {
while((sumLen += (locr=is.read(locbuf))) != fileLength) {
System.out.println("sumLen= "+sumLen);
osFile.write(locbuf, 0, locr);
}
}
===================================================
This works well.