I'm trying to send large files via socket. The program works fine for small files (such as html pages or pdf), but when i send files over 3/4 mb the output is always corrupted (viewing it with a text editor i noticed that the last few lines are always missing).
Here's the code of the server:
BufferedInputStream in = null;
FileOutputStream fout = null;
try {
server = new ServerSocket(port);
sock = server.accept();
in = new BufferedInputStream(sock.getInputStream());
setPerc(0);
received = 0;
int incByte = -1;
fout = new FileOutputStream(path+name, true);
long size = length;
do{
int buffSize;
if(size >= 4096){
buffSize = 4096;
}else{
buffSize = 1;
}
byte[] o = new byte[buffSize];
incByte = in.read(o, 0, buffSize);
fout.write(o);
received+=buffSize;
setPerc(calcPerc(received, length));
size -= buffSize;
//d("BYTE LETTI => "+incByte);
}while(size > 0);
server.close();
} catch (IOException e) {
e("Errore nella ricezione file: "+e);
}finally{
try {
fout.flush();
fout.close();
in.close();
} catch (IOException e) {
e("ERRORE INCOMINGFILE");
}
}
pr.release(port);
And here's the code of the client:
FileInputStream fin = null;
BufferedOutputStream out = null;
try {
sock = new Socket(host, port);
fin = new FileInputStream(file);
out = new BufferedOutputStream(sock.getOutputStream());
long size = file.length();
int read = -1;
do{
int buffSize = 0;
if(size >= 4096){
buffSize = 4096;
}else{
buffSize = (int)size;
}
byte[] o = new byte[buffSize];
for(int i = 0; i<o.length;i++){
o[i] = (byte)0;
}
read = fin.read(o, 0, buffSize);
out.write(o);
size -= buffSize;
//d("BYTE LETTI DAL FILE => "+read);
}while(size > 0);
} catch (UnknownHostException e) {
} catch (IOException e) {
d("ERRORE NELL'INVIO DEL FILE: "+e);
e.printStackTrace();
}finally{
try {
out.flush();
out.close();
fin.close();
} catch (IOException e) {
d("Errore nella chiusura dei socket invio");
}
}
i think it's something related with the buffer size, but i can't figure out what's wrong here.
This is incorrect:
byte[] o = new byte[buffSize];
incByte = in.read(o, 0, buffSize);
fout.write(o);
You are reading up to buffSize bytes and then writing exactly buffSize bytes.
You are doing the same thing at the other end as well.
You may be able to get away with this when reading from a file1, but when you read from a socket then a read is liable to give you a partially filled buffer, especially if the writing end can't always keep ahead of the reading end 'cos you are hammering the network with a large transfer.
The right way to do it is:
incByte = in.read(o, 0, buffSize);
fout.write(o, 0, incByte);
1 - It has been observed that when you read from a local file, a read call will typically give you all of the bytes that you requested (subject to the file size, etc). So, if you set buffSize to the length of the file, this code would probably work when reading from a local file. But doing this is a bad idea, because you are relying behaviour that is not guaranteed by either Java or a typical operating system.
You might have a problem e.g. here.
read = fin.read(o, 0, buffSize);
out.write(o);
Here read gives you the count of bytes you've actually just read.
On the next line you should write out only as many bytes as you've read.
In other words, you cannot expect the size of the file
you're reading to be multiple of your buffer size.
Review your server code too for the same issue.
The correct way to copy streams in Java is as follows:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
where count is an int, and buffer is a byte[] array of length > 0, typically 8k. You don't need to allocate byte arrays inside the loop, and you don't need a byte array of a specific size. Specifically, it's a complete waste of space to allocate a buffer as large as the file; it only works up to files of Integer.MAX_VALUE bytes, and it doesn't scale.
You do need to save the count returned by 'read()' and use it in the 'write()' method as shown above.
Related
Following is a code snippet which is supposed to read all images from image folder and encode them to a h.262 video and store in the sdcard. I followed the android documentation (get buffer, fill buffer, queue buffer for encoding, dequeue output buffer and then write to file). Problem is when I dequeue output buffer I get negative index whereas its supposed to return the index of output data. The output file is 0 Bytes and nothing is written into it.
I am pretty much very new in mediaCodec. Any suggestion would be appreciated.
MediaCodec mediaCodec=null;
byte[] input = new byte[2000];
BufferedOutputStream outputStream = null;
try {
//TODO
//adjust parameters by consulting with hari sir
mediaCodec = MediaCodec.createEncoderByType("video/avc");
MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/avc", 320, 240);
mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, 700000);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 10);
//not all phones support given color format, if color format is not supported app will crash with mediaCodec exception
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar);
mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);
mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
mediaCodec.start();
//after the mediaCodec is started we don't have ownership of input or output buffers
Log.i("Codecinfo",""+mediaCodec.getCodecInfo());
Log.i("Codecname",""+mediaCodec.getName());
}
catch (Exception e)
{
Log.e("ExceptionMediaCodec","Some exception in media codec");
}
//reached here
System.out.println("mediacodec info="+mediaCodec.getCodecInfo());
try {
File ff = new File(Environment.getExternalStorageDirectory(), "Download/video_encoded.264");
if (!ff.exists()) ff.createNewFile();
System.out.println("H.264 output file initialized");
outputStream = new BufferedOutputStream(new FileOutputStream(ff));
Log.i("H264 avc Encoder", "outputStream initialized");
} catch (Exception e){
e.printStackTrace();
}
String path = Environment.getExternalStorageDirectory().toString()+"/images";
File f = new File(path);
Log.i("ExternalFileInfo",path.toString());
//read image files onto an array
File[] files = f.listFiles();
System.out.println(files.getClass().getName());
int NUM_IMAGES = files.length;
String[] images = new String[NUM_IMAGES];
for (int i=0;i<NUM_IMAGES;i++)
images[i]=files[i].getName();
for (String eachimage: images) {
System.out.println(eachimage);
byte[] eachByte = eachimage.getBytes();
input = eachByte; //demo
System.out.println("input byte initialized"+input.toString());
try {
System.out.println("Following is the content of byte array input");
System.out.write(input);
}catch (Exception e)
{
e.printStackTrace();
}
}
//reached here
System.out.println("image byte size="+input.length);
//all images converted to bytearray
ByteBuffer[] outputBuffers = mediaCodec.getOutputBuffers();
//System.out.println("inputBuffers="+(inputBuffers));
//System.out.println("outputBuffers="+(outputBuffers));
//reached here
//returns the index of input buffer to be filled for encoding
int inputBufferIndex = mediaCodec.dequeueInputBuffer(-1); //-1 => wait indefinitely
System.out.println("inputBufferedIndex="+inputBufferIndex); //0
if (inputBufferIndex >= 0) {
ByteBuffer[] inputBuffers = mediaCodec.getInputBuffers();
ByteBuffer inputBuffer = inputBuffers[inputBufferIndex];
inputBuffer.clear();
System.out.println("input byte placed in input buffer");
inputBuffer.put(input);
System.out.println("inputBuffer after filling up" + inputBuffer);
mediaCodec.queueInputBuffer(inputBufferIndex, 0, input.length, System.nanoTime(), 0); //send each request with different timestamp
System.out.println("mediacodec input queued");
}
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
System.out.println("buffer info="+bufferInfo);
int outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, -1); //-ve value for indefinite waiting
//reached here
System.out.println("buffer info meta data=" + bufferInfo);
System.out.println("outputBufferedIndex=" + outputBufferIndex);
try {
while (outputBufferIndex >= 0) {
ByteBuffer outputBuffer = outputBuffers[outputBufferIndex];
byte[] outData = new byte[bufferInfo.size];
outputBuffer.get(outData);
outputStream.write(outData, 0, outData.length);
outputStream.flush();
Log.i("AvcEncoder", outData.length + " bytes written");
mediaCodec.releaseOutputBuffer(outputBufferIndex, false);
outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, -1);
}
} catch (Throwable t) {
t.printStackTrace();
}
try {
mediaCodec.stop();
mediaCodec.release();
outputStream.flush();
outputStream.close();
} catch (Exception e){
e.printStackTrace();
}
System.out.println("Mediacodec="+mediaCodec);
Please see the documentation for the MediaCodec.dequeueOutputBuffer() method, which says:
Returns the index of an output buffer that has been successfully decoded or one of the INFO_* constants.
The negative values are the INFO_* constants, which may be one of the following:
INFO_OUTPUT_BUFFERS_CHANGED
INFO_OUTPUT_FORMAT_CHANGED
INFO_TRY_AGAIN_LATER
The last one is not too probable because you're waiting indefinitely though.
Additionally: You can't always rely on waiting for one output buffer after you've given one single buffer as input. You need to feed input buffers as long as the encoder has got free input buffers, and consume whatever output buffers it gives you.
The last few output buffers might be output only once you signal that you won't be submitting any more input buffers, by setting the flag BUFFER_FLAG_END_OF_STREAM.
I try to write to a file, the data that I receive from a socket , I store the data in an array but when I write them, the file gets too big ...
I think it is caused by using a big array , as i don't know the length of the data stream...
But checking the method write it is stated that write(byte[] b) Writes b.length bytes from the specified byte array to this file output stream,
the write() method reads the length of the array but the length is 2000...
How can i know the length of the data that will be written?
...
byte[] Rbuffer = new byte[2000];
dis = new DataInputStream(socket.getInputStream());
dis.read(Rbuffer);
writeSDCard.writeToSDFile(Rbuffer);
...
void writeToSDFile(byte[] inputMsg){
File root = android.os.Environment.getExternalStorageDirectory();
File dir = new File (root.getAbsolutePath() + "/download");
if (!(dir.exists())) {
dir.mkdirs();
}
Log.d("WriteSDCard", "Start writing");
File file = new File(dir, "myData.txt");
try {
FileOutputStream f = new FileOutputStream(file, true);
f.write(inputMsg);
f.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
Log.i(TAG, "******* File not found. Did you" +
" add a WRITE_EXTERNAL_STORAGE permission to the manifest?");
} catch (IOException e) {
e.printStackTrace();
}
}
read() returns the number of bytes that were read, or -1. You are ignoring both possibilities, and assuming that it filled the buffer. All you have to do is store the result in a variable, check for -1, and otherwise pass it to the write() method.
Actually you should pass the input stream to your method, and use a loop after creating the file:
int count;
byte[] buffer = new byte[8192];
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Your statement in a now-deleted comment that a new input stream is created per packet is not correct.
we were given a few exercises in lab and one of these is to convert the file transferring method from FileInputStream to BufferedInputStream. It's a client sending a GET request to a web server, which sends the file requested.
I came up with a simple solution, and I just wanted to check if it's correct.
Original code:
try {
FileInputStream fis = new FileInputStream(req);
// req, String containing file name
byte[] data = new byte [fis.available()];
fis.read(data);
out.write(data); // OutputStream out = socket.getOutputStream();
} catch (FileNotFoundException e){
new PrintStream(out).println("404 Not Found");
}
My try:
try {
BufferedInputStream bis = new BufferedInputStream (new FileInputStream(req));
byte[] data = new byte[4];
while(bis.read(data) > -1) {
out.write(data);
data = new byte[4];
}
} catch (FileNotFoundException e){
new PrintStream(out).println("404 Not Found");
}
The file is a web page named index.html, which contains a simple html page.
I have to reallocate the array every time, because at the last execution of the while loop, if the file isn't a multiple of 4 in size, the data array will contain characters from the previous execution, which are shown in the browser.
I chose 4 as data size for debugging purposes.
Output is correct.
Is this a good solution or can I do better?
There's no need to re-create the byte array each time - just overwrite it. More importantly though, you have a conceptual mistake inside your loop. Each iteration just writes the array to the stream assuming it's all valid. If you examine BufferedInputStream#read's documentation you'll see it may not read enough data to fill the entire array, and will return the number of bytes it actually read. You should use this number to limit the amount of bytes you're writing:
while((int len = bis.read(data)) > -1) {
out.write(data, 0, len);
}
I suggest you close off your file once you are done. The BufferedInputStream uses an 8 KB buffer by default which you are reducing to a smaller buffer. A simpler solution is to copy 8 KB at a time and not use the added buffer
try (InputStream in = new FileInputStream(req)) {
byte[] data = new byte[8 << 10];
for (int len; (len = bis.read(data)) > -1; )
out.write(data, 0, len);
} catch (IOException e) {
out.write("404 Not Found\n".getBytes());
}
I've made a basic client server FTP program using sockets, but for some reason files are getting corrupted during the transfer. In the case below, I'm pushing a file to the server from the client. It almost works, since some files (such as a .png) transfer and open fine, but others (a .docx) don't. Any file that I transfer has a different MD5 to the one I sent.
Client code:
File file = null;
FTPDataBlock transferBlock;
int numBytesRead = 0;
int blockNumber = 1;
int blockSize = 1024;
byte[] block = new byte[blockSize];
fc = new JFileChooser();
// select file to upload
int returnVal = fc.showOpenDialog(Client.this);
if (returnVal == JFileChooser.APPROVE_OPTION) {
file = fc.getSelectedFile();
try {
// get total number of blocks and send to server
int totalNumBlocks = (int)Math.ceil((file.length()*1.0) / blockSize);
System.out.println("File length is: " + file.length());
FTPCommand c = new FTPCommand("PUSH", Integer.toString(totalNumBlocks));
oos = new ObjectOutputStream(sock.getOutputStream());
oos.writeObject(c);
oos.flush();
// send to server block by block
FileInputStream fin = new FileInputStream(file);
while ((numBytesRead = fin.read(block)) != -1){
transferBlock = new FTPDataBlock(file.getName(), blockNumber, block);
blockNumber++;
System.out.println("Sending block " + transferBlock.getBlockNumber() + " of " + totalNumBlocks);
oos = new ObjectOutputStream(sock.getOutputStream());
oos.writeObject(transferBlock);
oos.flush();
}
fin.close();
System.out.println("PUSH Complete");
// get response from server
ois = new ObjectInputStream(sock.getInputStream());
FTPResponse response = (FTPResponse)ois.readObject();
statusArea.setText(response.getResponse());
} catch (IOException | ClassNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
}
Server Code:
else if (cmd.getCommand().equals("PUSH")){
// get total number of file blocks
int totalNumBlocks = Integer.parseInt(cmd.getParameters());
// get first block
in = new ObjectInputStream(sock.getInputStream());
FTPDataBlock currentBlock = (FTPDataBlock)in.readObject();
// create file and write first block to file
File file = new File (workingDirectory + File.separator + currentBlock.getFilename());
FileOutputStream fOut = new FileOutputStream(file);
fOut.write(currentBlock.getData());
fOut.flush();
// get remaining blocks
while(currentBlock.getBlockNumber()+1 <= totalNumBlocks){
in = new ObjectInputStream(sock.getInputStream());
currentBlock = (FTPDataBlock)in.readObject();
fOut.write(currentBlock.getData());
fOut.flush();
}
fOut.close();
// send response
FTPResponse response = new FTPResponse("File Received OK");
out = new ObjectOutputStream(sock.getOutputStream());
out.writeObject(response);
}
FTPDataBlock class:
public class FTPDataBlock implements Serializable{
private static final long serialVersionUID = 1L;
private String filename;
private int blockNumber; // current block number
private byte[] data;
//constructors & accessors
}
I'm sure it's something small that I'm missing here. Any ideas?
This happened because the server was writing whole 1024 byte blocks to the file, even if there was less than 1024 bytes actually written to the block.
The solution (thanks to #kdgregory) was to use the return value of FileInputStream.read() to populate a new attribute in my FTPDataBlock class, int bytesWritten.
Then on the server side I could use:
FileOutputStream.write(currentBlock.getData(), 0, currentBlock.getBytesWritten());
to write the exact number of bytes to the file, instead of the whole block every time.
I think there may be a problem with the file extension. provide a option in the client side as:
FILE_TO_RECEIVED = JOptionPane.showInputDialog("Please enter the Drive followed by the file name to be saved. Eg: D:/xyz.jpg");
it is to help you to provide the correct file extension name.
then i think u should also provide the file size in client side like:
public final static int FILE_SIZE = 6022386;
and in then in the array block u used u can make the following changes as:
try {
sock = new Socket(SERVER, SOCKET_PORT);
byte [] mybytearray = new byte [FILE_SIZE];
InputStream is = sock.getInputStream();
fos = new FileOutputStream(FILE_TO_RECEIVED);
bos = new BufferedOutputStream(fos);
bytesRead = is.read(mybytearray,0,mybytearray.length);
current = bytesRead;
do {
bytesRead =
is.read(mybytearray, current, (mybytearray.length-current));
if(bytesRead >= 0) current += bytesRead;
} while(bytesRead > -1);
bos.write(mybytearray, 0 , current);
bos.flush();
}
I have a text file with a sequence of 4194304 letters ranging from A-D all on one line (4 MB).
How would I randomly point to a character and replace the following set of characters to another file that is 100 characters long and write it out to a file?
I'm actually currently able to do this, but I feel it's really inefficient when I iterate it several times.
Here's an illustration of what I mentioned above:
Link to Imageshack
Here's how I'm currently achieving this:
Random rnum = new Random();
FileInputStream fin = null;
FileOutputStream fout = null;
int count = 10000;
FileInputStream fin1 = null;
File file1 = new File("fileWithSet100C.txt");
int randChar = 0;
while(cnt > 0){
try {
int c = 4194304 - 100;
randChar = rnum.nextInt(c);
File file = new File("file.txt");
//seems inefficient to initiate these guys over and over
fin = new FileInputStream(file);
fin1 = new FileInputStream(file1);
//would like to remove this and have it just replace the original
fout = new FileOutputStream("newfile.txt");
int byte_read;
int byte_read2;
byte[] buffer = new byte[randChar];
byte[] buffer2 = new byte[(int)file1.length()]; //4m
byte_read = fin.read(buffer);
byte_read2 = fin1.read(buffer2);
fout.write(buffer, 0, byte_read);
fout.write(buffer2, 0, byte_read2);
byte_read = fin.read(buffer2);
buffer = new byte[4096]; //4m
while((byte_read = (fin.read(buffer))) != -1){
fout.write(buffer, 0, byte_read);
}
cnt--;
}
catch (...) {
...
}
finally {
...
}
try{
File file = new File("newfile.txt");
fin = new FileInputStream(file);
fout = new FileOutputStream("file.txt");
int byte_read;
byte[] buffer = new byte[4096]; //4m
byte_read = fin.read(buffer);
while((byte_read = (fin.read(buffer))) != -1){
fout.write(buffer, 0, byte_read);
}
}
catch (...) {
...
}
finally {
...
}
Thanks for reading!
EDIT:
For those curious, here's the code I used to solve the aforementioned problem:
String stringToInsert = "insertSTringHERE";
byte[] answerByteArray = stringToInsert.getBytes();
ByteBuffer byteBuffer = ByteBuffer.wrap(answerByteArray);
Random rnum = new Random();
randChar = rnum.nextInt(4194002); //4MB worth of bytes
File fi = new File("file.txt");
RandomAccessFile raf = null;
try {
raf = new RandomAccessFile(fi, "rw");
} catch (FileNotFoundException e1) {
// TODO error handling and logging
}
FileChannel fo = null;
fo = raf.getChannel();
// Move to the beginning of the file and write out the contents
// of the byteBuffer.
try {
outputFileChannel.position(randChar);
while(byteBuffer.hasRemaining()) {
fo.write(byteBuffer);
}
} catch (IOException e) {
// TODO error handling and logging
}
try {
outputFileChannel.close();
} catch (IOException e) {
// TODO error handling and logging
}
try {
randomAccessFile.close();
} catch (IOException e) {
// TODO error handling and logging
}
You probably want to use Java's random-access file features. Sun/Oracle has a Random Access Files tutorial that will probably be useful to you.
If you can't use Java 7, then look at RandomAccessFile which also has seek functionality and has existed since Java 1.0.
First off, for your files you could have the Files as global variables. This would all you to use the file when ever you needed without reading it again. Also note that if you keep making new files then you will lose the data that you have already acquired.
For example:
public class Foo {
// Gloabal Vars //
File file;
public Foo(String location) {
// Do Something
file = new File(location);
}
public add() {
// Add
}
}
Answering your question, I would first read both files and then make all the changes you want in memory. After you have made all the changes, I would then write the changes to the file.
However, if the files are very large, then I would make all the changes one by one on the disk... it will be slower, but you will not run out of memory this way. For what you are doing I doubt you could use a buffer to help counter how slow it would be.
My overall suggestion would be to use arrays. For example I would do the following...
public char[] addCharsToString(String str, char[] newChars, int index) {
char[] string = str.toCharArray();
char[] tmp = new char[string.length + newChars.length];
System.arraycopy(string, 0, tmp, 0, index);
System.arraycopy(newChars, index, tmp, index, newChars.length);
System.arraycopy(string, index + newChars.length, tmp, index + newChars.length, tmp.length - (newChars.length + index));
return tmp;
}
Hope this helps!