I need to upload a zip file to the URL using POST or another option. I followed the below method and successfully I have uploaded the file.
String diskFilePath="/tmp/test.zip;
String urlStr="https://<ip>/nfc/file.zip";
HttpsURLConnection conn = (HttpsURLConnection) new URL(urlStr).openConnection();
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setChunkedStreamingMode(CHUCK_LEN);
conn.setRequestMethod(put? "PUT" : "POST"); // Use a post method to write the file.
conn.setRequestProperty("Connection", "Keep-Alive");
conn.setRequestProperty("Content-Length", Long.toString(new File(diskFilePath).length()));
int i=0;
while(i<1)
{
continue;
}
BufferedOutputStream bos = new BufferedOutputStream(conn.getOutputStream());
BufferedInputStream diskis = new BufferedInputStream(new FileInputStream(diskFilePath));
int bytesAvailable = diskis.available();
int bufferSize = Math.min(bytesAvailable, CHUCK_LEN);
byte[] buffer = new byte[bufferSize];
long totalBytesWritten = 0;
while (true)
{
int bytesRead = diskis.read(buffer, 0, bufferSize);
if (bytesRead == -1)
{
//System.out.println("Total bytes written: " + totalBytesWritten);
break;
}
totalBytesWritten += bytesRead;
bos.write(buffer, 0, bufferSize);
bos.flush();
// System.out.println("Total bytes written: " + totalBytesWritten);
int progressPercent = (int) (((bytesAlreadyWritten + totalBytesWritten) * 100) / totalBytes);
}
Now my zip file is in a remote location and I need to upload the zip file without downloading to the local machine.
I need to pass this url “https:///file/test.zip” instead of “/tmp/test.zip”
For example, I execute the Program on machine A and the file to be uploaded is present in Machine B. Webserver is deployed in the machine B and url is exposed to download the zip file. Now I need to pass this ZIP file URL location to upload instead of the downloading the zip file to machine A and then pass to the upload URL.
Thanks,
Kalai
Not 100% what you mean by "without downloading to the local machine."
Here's how to avoid downloading the file into a temporary local file and then uploading that.
The basic approach is to read from one URLConnection (instead of the local file) and write to another URLConnection (like you do already).
Start by doing the request to sourceString, so
HttpsURLConnection source = (HttpsURLConnection) new URL("https://machine.B/path/to/file.zip").openConnection();
then keep everything until youset the Content-Length and replace that with
conn.setRequestProperty("Content-Length", source.getContentLength());
and then all you have to do is use
InputStream is = source.getInputStream();
instead of your diskis.
PS: I don't get the purpose of the .available logic, why not just use CHUNK_LEN for the buffer size?
PPS: also the while(i<0) loop can be removed ;-)
Related
In server side code, I have set buffer size and content length as File.length() and then Opened File using FileInputStream.
Later fetching output stream using HttpResponse.getOutputStream() and dumping bytes of data that is read using FileInputStream
I am using Apache Tomcat 7.0.52, Java 7
On Client
File Downloader.java
URL url = new URL("myFileURL");
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setDoInput(true);
con.setConnectTimeout(10000);
con.setReadTimeout(10000);
con.setRequestMethod("GET");
con.setUseCaches(false);
con.setRequestProperty("User-Agent", "Mozilla/5.0");
con.connect();
FileOutputStream fos = new FileOutputStream("filename");
if(con.getResponseCode()==200){
InputStream is = con.getInputStream();
int readVal;
while((readVal=is.read())!=-1) fos.write(readVal);
}
fos.flush()
fos.close();
So above code failed to download large file.
On client using Java 7
Can You try this
FileOutputStream outputStream = new FileOutputStream(fileName);
int bytesRead;
byte[] buffer = new byte[1024];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
Quoting from https://stackoverflow.com/a/45453874/4121845
Because you only want to write data that you actually read. Consider the case where the input consists of N buffers plus one byte. Without the len parameter you would write (N+1)*1024 bytes instead of N*1024+1 bytes. Consider also the case of reading from a socket, or indeed the general case of reading: the actual contract of InputStream.read() is that it transfers at least one byte, not that it fills the buffer. Often it can't, for one reason or another.
fos.flush();
} finally {
fos.close();
con.close();
}
I use HttpURLConnection to download files from a url.
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("GET");
con.setRequestProperty("Cache-Control", "no-cache");
int responseCode = con.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
try {
InputStream inputStream = con.getInputStream();
FileOutputStream outputStream = new FileOutputStream("C:\\programs\\TRYFILE.csv");
int bytesRead = -1;
byte[] buffer = new byte[4096];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
} catch(Exception e) {
} finally {
outputStream.close();
inputStream.close();
}
The code works to download small sized files (i.e. 25 KB). I didn't try to download large files (on the order of 100 MB), because the files from the particular URL are always small.
I want to know what happens if I try to download larger files with this code: will it continue to work or throw an exception? Do I need to implement code (utilizing, say, setConnectTimeout or setReadTimeout) for bigger files?
Is there a url you can suggest where I can try to download large file using this code?
As suggested in the comments, create a large file yourself. To serve it over HTTP, the easiest way is probably to run Python 3 from the directory where you put the file -
python -m http.server
which will start a server on 8000. See this blog post for more details, or check out the python documentation.
Then you can test this yourself.
i am writing a small android application which requires some data which is stored on my web server. The file is a .txt file curretly less than 1 MB. Is it advisable to set up a ftp server to get the data or can i just use a http get method to get the contents on a file. If i am using a http get can someone please tell me the java code required for this operation.
This is out of my head (so an error could have sneaked in):
URL url = new URL("http://www.yourserver.com/some/path");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
try {
InputStream in = new BufferedInputStream(urlConnection.getInputStream());
FileOutputStream out = new FileutputStream("/path/to/your/output/file");
byte[] buffer = new byte[16384];
int len;
while((len = in.read(buffer)) != -1){
out.write(buffer, 0, len);
}
finally {
urlConnection.disconnect();
}
I'm uploading an image file using a HttpURLConnection which takes about 3 seconds for a 5MB file with all the headers, but the moment I open an InputStream with .getInputStream(), the method takes about 8+ seconds to return a stream with. Which is an issue because it seems the upload progress bar gives a bad UX if I have multiple images to upload, they have a considerable pause between each upload, so the progress bar just stops for a couple of seconds between uploads. I've done some googling but no one else seems to have an issue with it?
Normally I would assume the server is slow, but seeing as uploading only takes a couple of seconds, downloading the word 'success' or 'fail' shouldn't really be that much of an issue!
Heres some code! Am I setting anything up wrong initially?
Note: This is also within an AsyncTask
ByteArrayInputStream fileInputStream = null;
try {
fileInputStream = new ByteArrayInputStream(dObject.Data);
} catch (Exception e) {
e.printStackTrace();
}
String lineEnd = "\r\n";
String twoHyphens = "--";
String boundary = "*****";
String Tag="3rd";
try
{
//------------------ CLIENT RE QUEST
Log.e(Tag,"Inside second Method");
// Open a HTTP connection to the URL
URL url = new URL(_urlString);
//connectURL is a URL object
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
// Allow Inputs
conn.setDoInput(true);
// Allow Outputs
conn.setDoOutput(true);
// Don't use a cached copy.
conn.setUseCaches(false);
// Use a post method.
conn.setRequestMethod("POST");
conn.setRequestProperty("Connection", "Keep-Alive");
conn.setRequestProperty("Content-Type", "multipart/form-data;boundary="+boundary);
DataOutputStream dos = new DataOutputStream( conn.getOutputStream() );
dos.writeBytes(twoHyphens + boundary + lineEnd);
//dos.writeBytes("Content-Disposition: form-data; name=\"uploadedfile\";filename=\"" + _fileLocation +"\"" + lineEnd);
dos.writeBytes("Content-Disposition: form-data; name=\"file\";filename=\"" + _fileLocation +"\"" + lineEnd);
dos.writeBytes(lineEnd);
Log.e(Tag,"Headers are written");
// create a buffer of maximum size
int bytesAvailable = fileInputStream.available();
int maxBufferSize = 1024;
int bufferSize = Math.min(bytesAvailable, maxBufferSize);
byte[] buffer = new byte[bufferSize];
// read file and write it into form...
int bytesRead = fileInputStream.read(buffer, 0, bufferSize);
while (bytesRead > 0) {
dos.write(buffer, 0, bufferSize);
//int value = (int)(((float)((float)totalRead / (float) fileSize)) * 100);
totalRead += bytesRead;
//Publish the progress out to be displayed
publishProgress(totalRead);
bytesAvailable = fileInputStream.available();
bufferSize = Math.min(bytesAvailable, maxBufferSize);
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
}
// send multipart form data necesssary after file data...
dos.writeBytes(lineEnd);
dos.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);
// close streams
Log.e(Tag,"File is written");
fileInputStream.close();
dos.flush();
Log.e("TIME", "PRE GETINPUTSTREAM");
InputStream is = conn.getInputStream();
Log.e("TIME", "POST GETINPUTSTREAM");
// retrieve the response from server
int ch;
//Build the respose and log
StringBuilder b =new StringBuilder();
while( ( ch = is.read() ) != -1 ){
b.append( (char)ch );
}
String s=b.toString();
Log.i("Response",s);
dos.close();
return;
}
catch (MalformedURLException ex)
{
ErrorHandler.get().e("3");
}
catch (IOException ioe)
{
ErrorHandler.get().e("2");
}
Normally I would assume the server is slow, but seeing as uploading only takes a couple of seconds, downloading the word 'success' or 'fail' shouldn't really be that much of an issue!
I suspect that it really is that the server is slow or overloaded.
The server could be queueing the HTTP requests and only processing a small number at a time in parallel.
Or it could have a bottleneck in some database activity that is performed before the response containing the file is written to the response.
Or it could be generating the file on the fly into an in-memory buffer (slow) and then streaming (fast) from the buffer to the HTTP response.
Or other explanations like this ...
(It is also theoretically possible that there is something funny going on that slows up the delivery of the request to the server. I would think this was unlikely though.)
Have you tried downloading the same file using a web browser? Do you get the same behaviour there?
In my case I found that getInputStream looks slow because this method initialize the ssl handshake (on a https URL call). After the first call, others calls are OK
I have this little piece of code below which uploads a file in java, the code functions correctly however it hangs for a long time when opening the output stream.
// open file to upload
InputStream filein = new FileInputStream("/path/to/file.txt");
// connect to server
URL url = new URL("ftp://user:pass#host/dir/file.txt");
URLConnection urlConn = url.openConnection();
urlConn.setDoOutput(true);
// write file
// HANGS FROM HERE
OutputStream ftpout = urlConn.getOutputStream();
// TO HERE for about 22 seconds
byte[] data = new byte[1024];
int len = 0;
while((len = filein.read(data)) > 0) {
ftpout.write(data,0, len);
}
// close file
filein .close();
ftpout.close();
In this example the URLConnection.getOutputStream() method hangs for about 22 seconds before continuing as normal, the file is successfully uploaded. The file is only 4 bytes in this case, just a text file with the word 'test' in it and the code hangs before the upload commences so its not because its taking time to upload the file.
This is only happening when connecting to one server, when I try a different server its as fast I could hope for which leads me to think it is a server configuration issue in which case this question may be more suited to server fault, however if I upload from an FTP client (in my case FileZilla) it works fine so it could be there is something I can do with the code to fix this.
Any ideas?
I have solved the problem by switching to use the Commons Net FTPClient which does not apear to have the same problems which changes the code to this below.
InputStream filein = new FileInputStream(new File("/path/to/file.txt"));
// create url
FTPClient ftp = new FTPClient();
ftp.connect(host);
ftp.login(user, pass);
int reply = ftp.getReplyCode();
if(!FTPReply.isPositiveCompletion(reply)) {
ftp.disconnect();
System.err.println("FTP server refused connection.");
return;
}
OutputStream ftpout = ftp.appendFileStream("text.txt");
// write file
byte[] data = new byte[1024];
int len = 0;
while((len = filein.read(data)) > 0) {
ftpout.write(data,0, len);
}
filein.close();
ftpout.close();
ftp.logout();