We Keep Coding

Remote file transfer using java nio

I am trying to use JAVA NIO to transfer a file from host A to client B without having to download the file locally and then giving the client B a link to download the file.
I am running a spark Apache framework and using maven project.
I mapped the request http://localhost:8080/download/hello in Spark using :
get("/download/:id",RequestHandler::downloadHandler);
Inside of the function is the code that downloads the file from :
"https://download.springsource.com/release/STS/3.8.1.RELEASE/dist/e4.6/spring-tool-suite-3.8.1.RELEASE-e4.6-linux-gtk-x86_64.tar.gz"
try {
URL url = new URL("https://download.springsource.com/release/STS/3.8.1.RELEASE/dist/e4.6/spring-tool-suite-3.8.1.RELEASE-e4.6-linux-gtk-x86_64.tar.gz");
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
int respCode = +httpURLConnection.getResponseCode();
System.out.println("response code : "+respCode);
if (respCode == HttpURLConnection.HTTP_OK){
String fileName = "";
String disposition = httpURLConnection.getHeaderField("Content-Disposition");
String contentType = httpURLConnection.getContentType();
int contentLength = httpURLConnection.getContentLength();
if (disposition != null) {
// extracts file name from header field
int index = disposition.indexOf("filename=");
if (index > 0) {
fileName = disposition.substring(index + 10,
disposition.length() - 1);
}
} else {
// extracts file name from URL
fileName = url.toString().substring(url.toString().lastIndexOf("/") + 1,
url.toString().length());
}
System.out.println("Content-Type = " + contentType);
System.out.println("Content-Disposition = " + disposition);
System.out.println("Content-Length = " + contentLength);
System.out.println("fileName = " + fileName);
httpURLConnection.disconnect();
System.out.println("other stuff : ");
System.out.println(url.getHost());
ReadableByteChannel readableByteChannel = Channels.newChannel(url.openStream());
FileOutputStream fileOutputStream = new FileOutputStream(fileName);
FileChannel fileChannel = fileOutputStream.getChannel();
fileChannel.transferFrom(readableByteChannel, 0, Long.MAX_VALUE);
fileOutputStream.close();
readableByteChannel.close();
}
} catch (IOException e) {
e.printStackTrace();
}
I fetch the filename and file size using httpURLConnection and then processed to download the file. what I am trying to do is, instead of downloading the file locally using fileChannel.transferFrom(readableByteChannel, 0, Long.MAX_VALUE) transfer the file directly to the client.
I did some research and I think it is possible with using Socketchannels but I didn't understand how it is supposed to work.
I also read this article
https://examples.javacodegeeks.com/core-java/nio/java-nio-large-file-transfer-tutorial/
and tried to understand the class Reciever, but it is still not clear to me how.
I would appreciate some guidance. Thank you

Downloaded files with whitespaces in path are damaged

I have following problem. I have to download pdf files from a server and some of them have whitespaces in their names. So every file will be downloaded, but those, which have whitespaces can not be opened.
If I access this files on the server via chrome, they open well (also with the whitespace in the url).
And what I am wondering about is, that java says the files will be downloaded. But when I try to open them in Acrobat Reader, it shows me an error message, that the files are damaged. Here is the sample of my code:
public static void downloadFile(String fileURL, String saveDir) throws IOException {
Authenticator.setDefault(new Authenticator() {
#Override
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication("*****", "*********".toCharArray());
}
});
final int BUFFER_SIZE = 4096;
URL url = new URL(fileURL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
String credentials = "ptt" + ":" + "ptt123";
String encoding = Base64.getEncoder().encodeToString(credentials.getBytes(StandardCharsets.UTF_8));
httpConn.setRequestProperty("Authorization", String.format("Basic %s", encoding));
int responseCode = 0;
responseCode = httpConn.getResponseCode();
// always check HTTP response code first
if (responseCode == HttpURLConnection.HTTP_OK) {
String fileName = "";
String disposition = httpConn.getHeaderField("Content-Disposition");
String contentType = httpConn.getContentType();
int contentLength = httpConn.getContentLength();
if (disposition != null) {
// extracts file name from header field
int index = disposition.indexOf("filename=");
if (index > 0) {
fileName = disposition.substring(index + 10,
disposition.length() - 1);
}
} else {
// extracts file name from URL
fileName = fileURL.substring(fileURL.lastIndexOf("/") + 1,
fileURL.length());
}
// opens input stream from the HTTP connection
InputStream inputStream = httpConn.getInputStream();
String saveFilePath = saveDir + File.separator + fileName;
// opens an output stream to save into file
FileOutputStream outputStream = new FileOutputStream(saveFilePath);
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();
System.out.println("File downloaded");
} else {
System.out.println("No file to download. Server replied HTTP code: " + responseCode);
}
httpConn.disconnect();
}
I also tried to replace the whitespace through "%20" in the fileUrl.
So what can be the problem? As I wrote above, the files without any whitespace can be opened after the the download without any problems.
I use Java 1.7.
Cheers,
Andrej

if fileName contains space then replace it to some other charecter. it may work, if not please let me know.
if(fileName.trim().contains(" "))
fileName.replace(" ","_");

URL url = new URL(URLEncoder.encode(fileUrl, "UTF-8"));

Parse upload to a class using REST java

I want to upload file to a class myclass in parse.com .Now, when I use the URL as https://api.parse.com/1/files/hello, I get the response message Created and the location of the file in the header. But, when I try to upload it to the class, I get the response message :- Bad Request and the header says that it is a 400 request. What am I doing wrong?
String name = "file.txt";
URL url = new URL("https://api.parse.com/1/classes/myclass/hello");
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
httpConn.setDoOutput(true);
httpConn.setRequestMethod("POST");
httpConn.setRequestProperty("X-Parse-Application-Id", "App_Id");
httpConn.setRequestProperty("X-Parse-REST-API-Key", "APp_KeY");
httpConn.setRequestProperty("Content-type", "text/plain");
OutputStream outputStream = httpConn.getOutputStream();
File uploadFile = new File("F:\\file.txt");
InputStream inputStream = new FileInputStream(uploadFile);
byte[] buffer = new byte[4096];
int bytesRead = -1;
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();
System.out.println("Response message : " + httpConn.getResponseMessage());

You can't upload using your class name I guess. What I do is first upload the file and then insert it into the class. I will be using Java since you have used java. So, first upload your file using https://api.parse.com/1/files/hello. Then you will get the location of the file in the header received. You can get it using httpConn.getResponseHeader("Location");. Now that will give you the entire url of the file. But you need only the name. You can extract it by this code:-
String ar[] = location.split("[/]");
String name = ar[ar.length - 1];
Now, insert it into your class. But before inserting, you have to make sure that you have a column that has data type of File in your class. Now, insert it into your class using this code :-
httpConn.setDoOutput(true);
httpConn.setRequestMethod("POST");
httpConn.setRequestProperty("X-Parse-Application-Id", "App_Id");
httpConn.setRequestProperty("X-Parse-REST-API-Key", "APp_KeY");
httpConn.setRequestProperty("Content-type", "application/json");
String json = "{\"myclass\":"
+ "{ "
+ "\"name\":\"" + name + "\","
+ "\"__type\": \"File\""
+ "}"
+ "}";
httpConn.getOutputStream().write(json.getBytes());
So, you have to make 2 requests in total.

Ruby : uploaded files suddenly become not found within hours in heroku

I have the following post code for Ruby Sinatra to upload image files :
post "/upload" do
File.open("public/uploads/" + params["image"][:filename], "wb") do |f|
f.write(params["image"][:tempfile].read)
end
end
And the following Java codes to upload the image file to example.com/upload :
private static String boundary;
private static final String LINE_FEED = "\r\n";
private static HttpURLConnection httpConn;
private static OutputStream outputStream;
private static PrintWriter writer;
public static void upload(String requestURL, String fieldName, File
uploadFile) throws IOException {
boundary = "===" + System.currentTimeMillis() + "===";
URL url = new URL(requestURL);
httpConn = (HttpURLConnection) url.openConnection();
httpConn.setUseCaches(false);
httpConn.setDoOutput(true);
httpConn.setDoInput(true);
httpConn.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
outputStream = httpConn.getOutputStream();
writer = new PrintWriter(new OutputStreamWriter(outputStream, "UTF-8"), true);
String fileName = uploadFile.getName();
writer.append("--" + boundary).append(LINE_FEED);
writer.append("Content-Disposition: form-data; name=\"" + fieldName + "\"; filename=\"" + fileName + "\"").append(LINE_FEED);
writer.append("Content-Type: "+ URLConnection.guessContentTypeFromName(fileName)).append(LINE_FEED);
writer.append("Content-Transfer-Encoding: binary").append(LINE_FEED);
writer.append(LINE_FEED).flush();
FileInputStream inputStream = new FileInputStream(uploadFile);
byte[] buffer = new byte[4096];
int bytesRead = -1;
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.flush();
inputStream.close();
writer.append("--" + boundary + "--").append(LINE_FEED);
writer.close();
httpConn.getInputStream();
httpConn.disconnect();
}
File uploadFile = new File("C:/myimage.png");
upload("http://www.example.com/upload", "image", uploadFile);
And my website is hosted at heroku .
After calling upload(), the image file is successfully uploaded, and i can access to it in example.com/upload/myimage.png
But the problem is : After some hours, when i check the url to see myimage.png, i got "Not Found" error ( 404 error in heroku logs )
Any ideas ?
Sorry for my bad english :|

You should not store files to heroku's local file system. From their docs:
Ephemeral filesystem
Each dyno gets its own ephemeral filesystem, with
a fresh copy of the most recently deployed code. During the dyno’s
lifetime its running processes can use the filesystem as a temporary
scratchpad, but no files that are written are visible to processes in
any other dyno and any files written will be discarded the moment the
dyno is stopped or restarted.
Instead of storing files locally it is recommended to upload files to AWS S3 or other cloud storage systems.

Image writing over URLConnection

I am trying to write an image over an HttpURLConnection.
I know how to write text but I am having real problems trying
to write an image
I have succeeded in writing to the local HD using ImageIO:
But I am trying to write Image by ImageIO on url and failed
URL url = new URL(uploadURL);
connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setDoInput(true);
connection.setUseCaches(false);
connection.setRequestProperty("Content-Type", "multipart/form-data;
boundary=" + boundary);
output = new DataOutputStream(connection.getOutputStream());
output.writeBytes("--" + boundary + "\r\n");
output.writeBytes("Content-Disposition: form-data; name=\"" + FIELD_NAME + "\";
filename=\"" + fileName + "\"\r\n");
output.writeBytes("Content-Type: " + dataMimeType + "\r\n");
output.writeBytes("Content-Transfer-Encoding: binary\r\n\r\n");
ImageIO.write(image, imageType, output);
the uploadURL is the url to an asp page on the server which will upload the image with the file name given in "content-Disposition: part.
now when I send this then asp page find the request and find the name of file. but does not find the file to be uploaded.
The problem is that when writing by ImageIO on URL what will the name of the file on which the ImageIO is writing,
So please help me how ImageIO will write an image on URLConnection and how can I know the name of the file which I have to use in the asp page to upload the file
Thanks for taking the time to read this post
Dilip Agarwal

First I believe that you should call io.flush() and then io.close() after writing image.
Second content type seems strange for me. It seems that you are trying to submit form while it is actually image. I do not know what does your asp expect but typically when I write code that should transfer file over HTTP I send appropriate content type, e.g. image/jpeg.
Here is for example code snippet I extracted from one small utility that I wrote and I am using during my current work:
URL url = new URL("http://localhost:8080/handler");
HttpURLConnection con = (HttpURLConnection)url.openConnection();
con.setDoInput(true);
con.setDoOutput(true);
con.setUseCaches(false);
con.setRequestProperty("Content-Type", "image/jpeg");
con.setRequestMethod("POST");
InputStream in = new FileInputStream("c:/temp/poc/img/mytest2.jpg");
OutputStream out = con.getOutputStream();
copy(in, con.getOutputStream());
out.flush();
out.close();
BufferedReader r = new BufferedReader(new InputStreamReader(con.getInputStream()));
// obviously it is not required to print the response. But you have
// to call con.getInputStream(). The connection is really established only
// when getInputStream() is called.
System.out.println("Output:");
for (String line = r.readLine(); line != null; line = r.readLine()) {
System.out.println(line);
}
I used here method copy() that I took from Jakarta IO utils. Here is the code for reference:
protected static long copy(InputStream input, OutputStream output)
throws IOException {
byte[] buffer = new byte[12288]; // 12K
long count = 0L;
int n = 0;
while (-1 != (n = input.read(buffer))) {
output.write(buffer, 0, n);
count += n;
}
return count;
}
Obviously the server side must be ready to read the image content directly from POST body.
I hope this helps.

The OP seems lost into oblivion but for the benefit of Mister Kite :
// main method
URL url = new URL(uploadURL);
connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true); // triggers "POST"
// connection.setDoInput(true); // only if needed
connection.setUseCaches(false); // dunno
final String boundary = Long.toHexString(System.currentTimeMillis());
connection.setRequestProperty("Content-Type", "multipart/form-data; boundary="
+ boundary);
output = new DataOutputStream(connection.getOutputStream());
try {
// image must be a File instance
flushMultiPartData(image, output, boundary);
} catch (IOException e) {
System.out.println("IOException in flushMultiPartData : " + e);
return;
}
// ...
private void flushMultiPartData(File file, OutputStream serverOutputStream,
String boundary) throws FileNotFoundException, IOException {
// SEE https://stackoverflow.com/a/2793153/281545
PrintWriter writer = null;
try {
// true = autoFlush, important!
writer = new PrintWriter(new OutputStreamWriter(serverOutputStream,
charsetForMultipartHeaders), true);
appendBinary(file, boundary, writer, serverOutputStream);
// End of multipart/form-data.
writer.append("--" + boundary + "--").append(CRLF);
} finally {
if (writer != null) writer.close();
}
}
private void appendBinary(File file, String boundary, PrintWriter writer,
OutputStream output) throws FileNotFoundException, IOException {
// Send binary file.
writer.append("--" + boundary).append(CRLF);
writer.append(
"Content-Disposition: form-data; name=\"binaryFile\"; filename=\""
+ file.getName() + "\"").append(CRLF);
writer.append("Content-Type: "
+ URLConnection.guessContentTypeFromName(file.getName()))
.append(CRLF);
writer.append("Content-Transfer-Encoding: binary").append(CRLF);
writer.append(CRLF).flush();
InputStream input = null;
try {
input = new FileInputStream(file);
byte[] buffer = new byte[1024];
for (int length = 0; (length = input.read(buffer)) > 0;) {
output.write(buffer, 0, length);
}
output.flush(); // Important! Output cannot be closed. Close of
// writer will close output as well.
} finally {
if (input != null) try {
input.close();
} catch (IOException logOrIgnore) {}
}
writer.append(CRLF).flush(); // CRLF is important! It indicates end of
// binary boundary.
}
You may want to add Gzip compression - see file corrupted when I post it to the servlet using GZIPOutputStream for a working class with or without Gzip. The ImageIO has no place here - just write the bytes past the wire and use ImageIO to your heart's content on the server. Based on #BalusC answer