I use HttpURLConnection to download files from a url.
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("GET");
con.setRequestProperty("Cache-Control", "no-cache");
int responseCode = con.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
try {
InputStream inputStream = con.getInputStream();
FileOutputStream outputStream = new FileOutputStream("C:\\programs\\TRYFILE.csv");
int bytesRead = -1;
byte[] buffer = new byte[4096];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
} catch(Exception e) {
} finally {
outputStream.close();
inputStream.close();
}
The code works to download small sized files (i.e. 25 KB). I didn't try to download large files (on the order of 100 MB), because the files from the particular URL are always small.
I want to know what happens if I try to download larger files with this code: will it continue to work or throw an exception? Do I need to implement code (utilizing, say, setConnectTimeout or setReadTimeout) for bigger files?
Is there a url you can suggest where I can try to download large file using this code?
As suggested in the comments, create a large file yourself. To serve it over HTTP, the easiest way is probably to run Python 3 from the directory where you put the file -
python -m http.server
which will start a server on 8000. See this blog post for more details, or check out the python documentation.
Then you can test this yourself.
Related
I need to upload a zip file to the URL using POST or another option. I followed the below method and successfully I have uploaded the file.
String diskFilePath="/tmp/test.zip;
String urlStr="https://<ip>/nfc/file.zip";
HttpsURLConnection conn = (HttpsURLConnection) new URL(urlStr).openConnection();
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setChunkedStreamingMode(CHUCK_LEN);
conn.setRequestMethod(put? "PUT" : "POST"); // Use a post method to write the file.
conn.setRequestProperty("Connection", "Keep-Alive");
conn.setRequestProperty("Content-Length", Long.toString(new File(diskFilePath).length()));
int i=0;
while(i<1)
{
continue;
}
BufferedOutputStream bos = new BufferedOutputStream(conn.getOutputStream());
BufferedInputStream diskis = new BufferedInputStream(new FileInputStream(diskFilePath));
int bytesAvailable = diskis.available();
int bufferSize = Math.min(bytesAvailable, CHUCK_LEN);
byte[] buffer = new byte[bufferSize];
long totalBytesWritten = 0;
while (true)
{
int bytesRead = diskis.read(buffer, 0, bufferSize);
if (bytesRead == -1)
{
//System.out.println("Total bytes written: " + totalBytesWritten);
break;
}
totalBytesWritten += bytesRead;
bos.write(buffer, 0, bufferSize);
bos.flush();
// System.out.println("Total bytes written: " + totalBytesWritten);
int progressPercent = (int) (((bytesAlreadyWritten + totalBytesWritten) * 100) / totalBytes);
}
Now my zip file is in a remote location and I need to upload the zip file without downloading to the local machine.
I need to pass this url “https:///file/test.zip” instead of “/tmp/test.zip”
For example, I execute the Program on machine A and the file to be uploaded is present in Machine B. Webserver is deployed in the machine B and url is exposed to download the zip file. Now I need to pass this ZIP file URL location to upload instead of the downloading the zip file to machine A and then pass to the upload URL.
Thanks,
Kalai
Not 100% what you mean by "without downloading to the local machine."
Here's how to avoid downloading the file into a temporary local file and then uploading that.
The basic approach is to read from one URLConnection (instead of the local file) and write to another URLConnection (like you do already).
Start by doing the request to sourceString, so
HttpsURLConnection source = (HttpsURLConnection) new URL("https://machine.B/path/to/file.zip").openConnection();
then keep everything until youset the Content-Length and replace that with
conn.setRequestProperty("Content-Length", source.getContentLength());
and then all you have to do is use
InputStream is = source.getInputStream();
instead of your diskis.
PS: I don't get the purpose of the .available logic, why not just use CHUNK_LEN for the buffer size?
PPS: also the while(i<0) loop can be removed ;-)
I have a problem with downloading a zip file from an url.
It works well with firefox but with my app I have a 404.
Here is my code
URL url = new URL(reportInfo.getURI().toString());
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
// Check for errors
int responseCode = con.getResponseCode();
InputStream inputStream;
if (responseCode == HttpURLConnection.HTTP_OK) {
inputStream = con.getInputStream();
} else {
inputStream = con.getErrorStream();
}
OutputStream output = new FileOutputStream("test.zip");
// Process the response
BufferedReader reader;
String line = null;
reader = new BufferedReader(new InputStreamReader(inputStream));
while ((line = reader.readLine()) != null) {
output.write(line.getBytes());
}
output.close();
inputStream.close();
Any idea ?
In Java 7, the easiest way to save a URL to a file is:
try (InputStream stream = con.getInputStream()) {
Files.copy(stream, Paths.get("test.zip"));
}
As for why you're getting a 404 - that hard to tell. You should check the value of url, which as greedybuddha says, you should get via URI.getURL(). But it's also possible that the server is using a user agent check or something similar to determine whether or not to give you the resource. You could try with something like cURL to fetch in programmatic way but without having to write any code yourself.
However, there another problem looming. It's a zip file. That's binary data. But you're using InputStreamReader, which is designed for text content. Don't do that. You should never use a Reader for binary data. Just use the InputStream:
byte[] buffer = new byte[8 * 1024]; // Or whatever
int bytesRead;
while ((bytesRead = inputStream.read(buffer)) > 0) {
output.write(buffer, 0, bytesRead);
}
Note that you should close the streams in finally blocks, or use the try-with-resources statement if you're using Java 7.
After almost 2 workdays of Googling and trying several different possibilities I found throughout the web, I'm asking this question here, hoping that I might finally get an answer.
First of all, here's what I want to do:
I'm developing a client and a server application with the purpose of exchanging a lot of large files between multiple clients on a single server. The client is developed in pure Java (JDK 1.6), while the web application is done in Grails (2.0.0).
As the purpose of the client is to allow users to exchange a lot of large files (usually about 2GB each), I have to implement it in a way, so that the uploads are resumable, i.e. the users are able to stop and resume uploads at any time.
Here's what I did so far:
I actually managed to do what I wanted to do and stream large files to the server while still being able to pause and resume uploads using raw sockets. I would send a regular request to the server (using Apache's HttpClient library) to get the server to send me a port that was free for me to use, then open a ServerSocket on the server and connect to that particular socket from the client.
Here's the problem with that:
Actually, there are at least two problems with that:
I open those ports myself, so I have to manage open and used ports myself. This is quite error-prone.
I actually circumvent Grails' ability to manage a huge amount of (concurrent) connections.
Finally, here's what I'm supposed to do now and the problem:
As the problems I mentioned above are unacceptable, I am now supposed to use Java's URLConnection/HttpURLConnection classes, while still sticking to Grails.
Connecting to the server and sending simple requests is no problem at all, everything worked fine. The problems started when I tried to use the streams (the connection's OutputStream in the client and the request's InputStream in the server). Opening the client's OutputStream and writing data to it is as easy as it gets. But reading from the request's InputStream seems impossible to me, as that stream is always empty, as it seems.
Example Code
Here's an example of the server side (Groovy controller):
def test() {
InputStream inStream = request.inputStream
if(inStream != null) {
int read = 0;
byte[] buffer = new byte[4096];
long total = 0;
println "Start reading"
while((read = inStream.read(buffer)) != -1) {
println "Read " + read + " bytes from input stream buffer" //<-- this is NEVER called
}
println "Reading finished"
println "Read a total of " + total + " bytes" // <-- 'total' will always be 0 (zero)
} else {
println "Input Stream is null" // <-- This is NEVER called
}
}
This is what I did on the client side (Java class):
public void connect() {
final URL url = new URL("myserveraddress");
final byte[] message = "someMessage".getBytes(); // Any byte[] - will be a file one day
HttpURLConnection connection = url.openConnection();
connection.setRequestMethod("GET"); // other methods - same result
// Write message
DataOutputStream out = new DataOutputStream(connection.getOutputStream());
out.writeBytes(message);
out.flush();
out.close();
// Actually connect
connection.connect(); // is this placed correctly?
// Get response
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = null;
while((line = in.readLine()) != null) {
System.out.println(line); // Prints the whole server response as expected
}
in.close();
}
As I mentioned, the problem is that request.inputStream always yields an empty InputStream, so I am never able to read anything from it (of course). But as that is exactly what I'm trying to do (so I can stream the file to be uploaded to the server, read from the InputStream and save it to a file), this is rather disappointing.
I tried different HTTP methods, different data payloads, and also rearranged the code over and over again, but did not seem to be able to solve the problem.
What I hope to find
I hope to find a solution to my problem, of course. Anything is highly appreciated: hints, code snippets, library suggestions and so on. Maybe I'm even having it all wrong and need to go in a totally different direction.
So, how can I implement resumable file uploads for rather large (binary) files from a Java client to a Grails web application without manually opening ports on the server side?
HTTP GET method have special headers for range retrieval: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35 It's used by most downloaders to do resumable download from server.
As I understand, there are no standard practice for using this headers for POST/PUT request, but it's up to you, right? You can make pretty standard Grails controller, that will accept standard http upload, with header like Range: bytes=500-999. And controller should put this 500 uploaded bytes from client into file, starting at position 500
At this case you don't need to open any socket, and make own protocols, etc.
P.S. 500 bytes is just a example, probably you're using much bigger parts.
Client Side Java Programming:
public class NonFormFileUploader {
static final String UPLOAD_URL= "http://localhost:8080/v2/mobileApp/fileUploadForEOL";
static final int BUFFER_SIZE = 4096;
public static void main(String[] args) throws IOException {
// takes file path from first program's argument
String filePath = "G:/study/GettingStartedwithGrailsFinalInfoQ.pdf";
File uploadFile = new File(filePath);
System.out.println("File to upload: " + filePath);
// creates a HTTP connection
URL url = new URL(UPLOAD_URL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
httpConn.setDoOutput(true);
httpConn.setRequestMethod("POST");
// sets file name as a HTTP header
httpConn.setRequestProperty("fileName", uploadFile.getName());
// opens output stream of the HTTP connection for writing data
OutputStream outputStream = httpConn.getOutputStream();
// Opens input stream of the file for reading data
FileInputStream inputStream = new FileInputStream(uploadFile);
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead = -1;
while ((bytesRead = inputStream.read(buffer)) != -1) {
System.out.println("bytesRead:"+bytesRead);
outputStream.write(buffer, 0, bytesRead);
outputStream.flush();
}
System.out.println("Data was written.");
outputStream.flush();
outputStream.close();
inputStream.close();
int responseCode = httpConn.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
// reads server's response
BufferedReader reader = new BufferedReader(new InputStreamReader(
httpConn.getInputStream()));
String response = reader.readLine();
System.out.println("Server's response: " + response);
} else {
System.out.println("Server returned non-OK code: " + responseCode);
}
}
}
Server Side Grails Programme:
Inside the controller:
def fileUploadForEOL(){
def result
try{
result = mobileAppService.fileUploadForEOL(request);
}catch(Exception e){
log.error "Exception in fileUploadForEOL service",e
}
render result as JSON
}
Inside the Service Class:
def fileUploadForEOL(request){
def status = false;
int code = 500
def map = [:]
try{
String fileName = request.getHeader("fileName");
File saveFile = new File(SAVE_DIR + fileName);
System.out.println("===== Begin headers =====");
Enumeration<String> names = request.getHeaderNames();
while (names.hasMoreElements()) {
String headerName = names.nextElement();
System.out.println(headerName + " = " + request.getHeader(headerName));
}
System.out.println("===== End headers =====\n");
// opens input stream of the request for reading data
InputStream inputStream = request.getInputStream();
// opens an output stream for writing file
FileOutputStream outputStream = new FileOutputStream(saveFile);
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead = inputStream.read(buffer);
long count = bytesRead
while(bytesRead != -1) {
outputStream.write(buffer, 0, bytesRead);
bytesRead = inputStream.read(buffer);
count += bytesRead
}
println "count:"+count
System.out.println("Data received.");
outputStream.close();
inputStream.close();
System.out.println("File written to: " + saveFile.getAbsolutePath());
code = 200
}catch(Exception e){
mLogger.log(java.util.logging.Level.SEVERE,"Exception in fileUploadForEOL",e);
}finally{
map <<["code":code]
}
return map
}
I have tried with above code it is worked for me(only for file size 3 to 4MB, but for small size files some bytes of code missing or not even coming but in request header content-length is coming, not sure why it is happening.)
i am writing a small android application which requires some data which is stored on my web server. The file is a .txt file curretly less than 1 MB. Is it advisable to set up a ftp server to get the data or can i just use a http get method to get the contents on a file. If i am using a http get can someone please tell me the java code required for this operation.
This is out of my head (so an error could have sneaked in):
URL url = new URL("http://www.yourserver.com/some/path");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
try {
InputStream in = new BufferedInputStream(urlConnection.getInputStream());
FileOutputStream out = new FileutputStream("/path/to/your/output/file");
byte[] buffer = new byte[16384];
int len;
while((len = in.read(buffer)) != -1){
out.write(buffer, 0, len);
}
finally {
urlConnection.disconnect();
}
I'm using this Java code to download a file from the Internet:
String address = "http://melody.syr.edu/pzhang/publications/AMCIS99_vonDran_Zhang.pdf";
URL url = new URL(address);
System.out.println("Opening connection to " + address + "...");
URLConnection urlC = url.openConnection();
urlC.setRequestProperty("User-Agent", "");
urlC.connect();
InputStream is = urlC.getInputStream();
FileOutputStream fos = null;
fos = new FileOutputStream("myFileName");
int oneChar, count = 0;
while ((oneChar = is.read()) != -1) {
System.out.print((char)oneChar);
fos.write(oneChar);
count++;
}
is.close();
fos.close();
System.out.println(count + " byte(s) copied");
I'd like to know if there is a way for me to download only a part of a file.
For example, for a 5MB file to download the last 2MB.
If the server supports it (and HTTP 1.1 servers should), you can use range requests:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35
Also, reading one character at a time is hugely inefficient - you should be reading in blocks, say 4, 16 or 32 KB.
Please have a look at Java: resume Download in URLConnection