I'm looking for an efficient way to transfer files between client and server processes using TCP in Java. My server code looks something like this:
socket = serverSocket.accept();
InputStream is = socket.getInputStream();
OutputStream os = socket.getOutputStream();
FileInputStream fis = new FileInputStream(new File(filename));
I'm just unsure of how to proceed. I know I want to read bytes from fis and then write them to os, but I'm unsure about the best way to read and write bytes using byte streams in Java. I'm only familiar with writing/reading text using Writers and Readers. Can anyone tell me the appropriate way to do this? What should I wrap os and fis in (if anything) and how do I keep reading bytes until the end of file without a hasNext() method (or equivalent)
You could do something like:
byte[] contents = new byte[BUFFER_SIZE];
int numBytes =0;
while((numBytes = is.read(contents))>0){
os.write(contents,0,numBytes);
}
You could use Apache's IOUtils.copy(in, out) or
import org.apache.commons.fileupload.util.Streams;
...
Streams.copy(in, out, false);
Inspecting the source might prove interesting. ( http://koders.com ?)
There is the java.nio.Channel with a transferTo method, with mixed opinions in the community wether better for smaller/larger files.
A simple block wise copy between Input/OutputStream would be okay. You could wrap it in buffered streams.
Related
So I have created my own personal HTTP Server in Java from scratch.
So far it is working fine but with one major flaw.
When I try to pass big files to the browser I get a Java Heap Space error. I know how to fix this error through the JVM but I am looking for the long term solution for this.
//declare an integer for the byte length of the file
int length = (int) f.length();
//start the fileinput stream.
FileInputStream fis = new FileInputStream(f);
//byte array with the length of the file
byte[] bytes = new byte[length];
//write the file until the bytes is empty.
while ((length = fis.read(bytes)) != -1 ){
write(bytes, 0, length);
}
flush();
//close the file input stream
fis.close();
This way sends the file to the browser successfully and streams it perfectly but the issue is, because I am creating a byte array with the length of the file. When the file is very big I get the Heap Space error.
I have eliminated this issue by using a buffer as shown below and I dont get Heap Space errors anymore. BUT the way shown below does not stream the files in the browser correctly. It's as if the file bytes are being shuffled and are being sent to the browser all together.
final int bufferSize = 4096;
byte buffer[] = new byte[bufferSize];
FileInputStream fis = new FileInputStream(f);
BufferedInputStream bis = new BufferedInputStream(fis);
while ( true )
{
int length = bis.read( buffer, 0, bufferSize );
if ( length < 0 ) break;
write( buffer, 0, length );
}
flush();
bis.close();
fis.close();
NOTE1:
All the correct Response Headers are being sent perfectly to the browser.
Note2:
Both ways work perfectly on a computer browser but only the first way works on a smartphone's browser (but sometimes it gives me Heap Space error).
If someone knows how to correctly send files to a browser and stream them correctly I would be a very very happy man.
Thank you in advance! :)
When reading from a BufferedInputStream you can allow its' buffer to handle the buffering, there is no reason to read everything into a byte[] (and certainly not a byte[] of the entire File). Read one byte at a time, and rely on the internal buffer of the stream. Something like,
FileInputStream fis = new FileInputStream(f);
BufferedInputStream bis = new BufferedInputStream(fis);
int abyte;
while ((abyte = bis.read()) != -1 ){
write(abyte);
}
Emm... As I can see it, you try to use chunks in your code anyway,
as I can remember, even the apache HttpClient+FileUpload solution has file size limit about <=2.1GB or something (correct me if I am wrong) so it is a bit hard thing...
I haven't tried the solution yet but as a test you can use java.io.RandomAccessFile in combination with File(Input/Output)Stream on the client and server not to read and write the whole file at a time but sequence of lets say <=30MB blocks for example to avoid the annoying outofmemory errors ; An example of using RandomAccessFile can be found here https://examples.javacodegeeks.com/core-java/io/randomaccessfile/java-randomaccessfile-example/
But still you give less details :( I mean is your client suppose to be a common Java application or not?
If you have some additional information please let me know
Good luck :)
I was trying to read from file and then write to other file. I use code bellow to do so.
byte[] bytes = Files.readAllBytes(file1);
Writer Writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file2), "UTF-8"));
for(int i=0;i<bytes.length;i++)
Writer.write(bytes[i]);
Writer.close();
But when I change file1 to picture.png and file2 to picture2.png, this method doesn't work and I can't open picture2.png using image viewer.
What have I done wrong?
Writers are for writing text, possibly in different formats (ie utf-8 / 16, etc). For writing raw bytes, don't use writers. Just use (File)OutputStreams.
It is truly as simple as
byte[] bytes = ...;
FileOutputStream fos = ...;
fos.write(bytes);
The other answers explain why what you have potentially fails.
I'm curious why you're already using one Java NIO method, but not others? The library already has methods to do this for you.
byte[] bytes = Files.readAllBytes(file1);
Files.write(file2, bytes, StandardOpenOption.CREATE_NEW); // or relevant OpenOptions
or
FileOutputStream out = new FileOutputStream(file2); // or buffered
Files.copy(file1, out);
out.close();
or
Files.copy(file1, file2, options);
The problem is that Writer.write() doesn't take a byte. It takes a char, which is variable size, and often bigger than one byte.
But once you've got the whole thing read in as a byte[], you can just use Files.write() to send the whole array to a file in much the same way that you read it in:
Files.write(filename, bytes);
This is the more modern NIO idiom, rather than using an OutputStream.
It's worth reading the tutorial.
I have text file on my ftp server. I am trying to write into this file but couldn't. This is my code.
URL url = new URL("ftp://username:pass#thunder.cise.ufl.edu/public/foler/a.txt;type=i");
URLConnection urlc = url.openConnection();
OutputStream os = urlc.getOutputStream(); // To upload
OutputStream buffer = new BufferedOutputStream(os);
ObjectOutput output = new ObjectOutputStream(buffer);
output.writeChars("hello");
buffer.close();
os.close();
output.close();
ObjectOutputStream class is intended to write object data so it can be reconstructed by ObjectInputStream (see here). It's not for writing textual files. If all you need is writing String to stream better use PrintStream
URL url = new URL("ftp://username:pass#thunder.cise.ufl.edu/public/foler/a.txt;type=i");
URLConnection urlc = url.openConnection();
OutputStream os = urlc.getOutputStream(); // To upload
OutputStream buffer = new BufferedOutputStream(os);
PrintStream output = new PrintStream(buffer);
output.print("hello");
buffer.close();
os.close();
output.close();
what library do you use?
I think you must use a right java library when connecting to FTP
I used this one in my previous projects
ApacheCommons FTPClient
feel free to ask if you have problems using the above library.
Take a look at the question Uploading to FTP using Java and look at the answer by user Loša.
The only thing that Loša's answer is missing is the definition of the var BUFFER_SIZE as
final int BUFFER_SIZE = 1024; // or whatever size you think it should be
and importing libraries and basic class definition for what you're doing.
Some simple searching here, or via DuckDuckGo or Google would have found what you're looking for.
Also, you aren't asking a question so much as saying "This doesn't work and I don't know why. Fix it for me."
Consider these two functions:
Function A takes inputStream as parameter.
public void processStream(InputStream stream)
{
//Do process routine
}
Function B loads a file content to pass it to Function A as InputStream.
pulic void loadFile()
{
File file =new File("c:\\file.txt");
//Pass file as InputStream
}
How can I pass file from Function B to Function A as InputStream without reading it on first hand?
I did something like this:
File file = new File("c:\\file.txt");
DataInputStream stream= new DataInputStream(new FileInputStream(file));
This generated the exception below:
java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: java.io.DataInputStream
EDIT:
loadFile() is passing the InputStream as RMI response.
The following should work just fine
processStream(new FileInputStream(file));
You should only not attempt to serialize an InputStream instance by ObjectOutputStream like as
objectOutputStream.writeObject(inputStream);
which you're apparently doing in processStream() method. That's namely exactly what the exception is trying to tell you. How to solve it properly depends on the sole functional requirement which you omitted from the question.
Update as per the comment
I am passing the InputStream as an RMI response.
There's the problem. You cannot pass non-serializable objects around as RMI response, let alone unread streams. You need to read the InputStream into a ByteArrayOutputStream the usual IO way and then use its toByteArray() to get a byte[] out of it and pass that instead. Something like:
InputStream input = new FileInputStream(file);
ByteArrayOutputStream output = new ByteArrayOutputStream();
byte[] buffer = new byte[8192];
for (int length = 0; (length = input.read(buffer)) > 0;) {
output.write(buffer, 0, length);
}
byte[] bytes = output.toByteArray(); // Pass that instead to RMI response.
Be careful with large files though. Every byte of a byte[] eats one byte of JVM's memory.
That exception seems to indicate that you are calling the processStream method on a remote object using something like RMI? if that is the case, you will need to re-visit what you are doing. sending streams of data over RMI is not an easy thing to do. if you are guaranteed to be using small files, you could copy the file data to a byte[] and pass that to the remote method call. if you need to process larger files, however, that will most likely cause memory issues on the client and/or server. in that case, you should use something like rmiio, which provides utilities for streaming data over RMI.
You could just pass the FileInputStream ?
processStream(new FileInputStream(yourFile));
The reason you are getting the exception is because DataInputStream is intended to read primitive Java types
I was handed some data in a file with an .dat extension. I need to read this data in a java program and build the data into some objects we defined. I tried the following, but it did not work
FileInputStream fstream = new FileInputStream("news.dat");
BufferedReader br = new BufferedReader(new InputStreamReader(fstream));
Could someone tell me how to do this in java?
What kind of file is it? Is it a binary file which contains serialized Java objects? If so, then you rather need ObjectInputStream instead of DataInputStream to read it.
FileInputStream fis = new FileInputStream("news.dat");
ObjectInputStream ois = new ObjectInputStream(fis);
Object object = ois.readObject();
// ...
(don't forget to properly handle resources using close() in finally, but that's beyond the scope of this question)
See also:
Basic serialization tutorial
A .dat file is usually a binary file, without any specific associated format. You can read the raw bytes of the file in a manner similar to what you posted - but you will need to interpret these bytes according to the underlying format. In particular, when you say "open" the file, what exactly do you want to happen in Java? What kind of objects do you want to be created? How should the stream of bytes map to these objects?
Once you know this, you can either write this layer yourself or use an existing API (assuming it's a standard format).
For reference, your example doesn't work because it assumes that the binary format is a character representation in the platform's default charset (as per the InputStreamReader constructor). And as you say it's binary, this will fail to convert the binary to a stream of characters (since, after all, it's not).
// BufferedInputStream not strictly needed, but much more efficient than reading
// one byte at a time
BufferedInputStream in = new BufferedInputStream (new FileInputStream("news.dat"));
This will give you a buffered stream which will return the raw bytes of the file; you can now either read and process them yourself, or pass this input stream to some library API that will create appropriate objects for you (if such a library exists).
That entirely depends on what sort of file the .dat is. Unfortunately, .dat is often used as a generic extension for a data file. It could be binary, in which case you could use FileInputStream fstream = new FileInputStream(new File("news.dat")); and call read() to get bytes from the file, or text, in which case you could use BufferedReader buff = new BufferedInputReader(new FileInputStream(new File("news.dat"))); and call readLine() to get each line of text. [edit]Or it could be Java objects in which case what BalusC said.[/edit]
In both cases, you'd then need to know what format the file was in to divide things up and get meaning from it, although this would be much easier if it was text as it could be done by inspection.
Please try the below code:
FileReader file = new FileReader(new File("File.dat"));
BufferedReader br = new BufferedReader(file);
String temp = br.readLine();
while (temp != null) {
temp = br.readLine();
System.out.println(temp);
}
A better way would be to use try-with-resources so that you would not have to worry about closing the resources.
Here is the code.
FileInputStream fis = new FileInputStream("news.dat");
try(ObjectInputStream objectstream = new ObjectInputStream(fis)){
objectstream.readObject();
}
catch(IOException e){
//
}