I tried to create byte array blocks from file whil the process was still using the file for writing. Actually I am storing video into file and I would like to create chunks from the same file while recording.
The following method was supposed to read blocks of bytes from file:
private byte[] getBytesFromFile(File file) throws IOException{
InputStream is = new FileInputStream(file);
long length = file.length();
int numRead = 0;
byte[] bytes = new byte[(int)length - mReadOffset];
numRead = is.read(bytes, mReadOffset, bytes.length - mReadOffset);
if(numRead != (bytes.length - mReadOffset)){
throw new IOException("Could not completely read file " + file.getName());
}
mReadOffset += numRead;
is.close();
return bytes;
}
But the problem is that all array elements are set to 0 and I guess it is because the writing process locks the file.
I would bevery thankful if anyone of you could show any other way to create file chunks while writing into file.
Solved the problem:
private void getBytesFromFile(File file) throws IOException {
FileInputStream is = new FileInputStream(file); //videorecorder stores video to file
java.nio.channels.FileChannel fc = is.getChannel();
java.nio.ByteBuffer bb = java.nio.ByteBuffer.allocate(10000);
int chunkCount = 0;
byte[] bytes;
while(fc.read(bb) >= 0){
bb.flip();
//save the part of the file into a chunk
bytes = bb.array();
storeByteArrayToFile(bytes, mRecordingFile + "." + chunkCount);//mRecordingFile is the (String)path to file
chunkCount++;
bb.clear();
}
}
private void storeByteArrayToFile(byte[] bytesToSave, String path) throws IOException {
FileOutputStream fOut = new FileOutputStream(path);
try {
fOut.write(bytesToSave);
}
catch (Exception ex) {
Log.e("ERROR", ex.getMessage());
}
finally {
fOut.close();
}
}
If it were me, I would have it chunked by the process/thread writing to the file. This is how Log4j seems to do it, at any rate. It should be possible to make an OutputStream which automatically starts writing to a new file every N bytes.
Related
I am trying to transfer a .mp4 file using WebRTC and it's DataChannel. In order to do that I am breaking the file into chunks like below:
FileInputStream is = new FileInputStream(file);
byte[] chunk = new byte[260000];
int chunkLen = 0;
sentFileByte = new ArrayList<>();
while ((chunkLen = is.read(chunk)) != -1) {
sentFileByte.add(chunk);
}
After that, sending the chunks by index like:
byte[] b = sentFileByte.get(index);
ByteBuffer bb = ByteBuffer.wrap(b);
bb.put(b);
bb.flip();
dataChannel.send(new DataChannel.Buffer(bb, true));
On the receiver end I am receiving the chunks and adding it to an Arraylist
receivedFileByteArr.add(chunkByteArr);
After receiving all the chunks successfully I am trying to convert these in to a file like below:
String path = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS).getAbsolutePath() + "/" + fileName;
File file = new File(path);
try {
FileOutputStream fileOutputStream = new FileOutputStream(file);
for (int i = 0; i < receivedFileByteArr.size(); i++) {
fileOutputStream.write(receivedFileByteArr.get(i));
}
fileOutputStream.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
After completing all these steps, file is created successfully. File size is also same. But problem is the file is not playable in any video player. I guess I am making some mistake on FileInputStream and FileOutputStream. I need help to fix this error.
I try to write to a file, the data that I receive from a socket , I store the data in an array but when I write them, the file gets too big ...
I think it is caused by using a big array , as i don't know the length of the data stream...
But checking the method write it is stated that write(byte[] b) Writes b.length bytes from the specified byte array to this file output stream,
the write() method reads the length of the array but the length is 2000...
How can i know the length of the data that will be written?
...
byte[] Rbuffer = new byte[2000];
dis = new DataInputStream(socket.getInputStream());
dis.read(Rbuffer);
writeSDCard.writeToSDFile(Rbuffer);
...
void writeToSDFile(byte[] inputMsg){
File root = android.os.Environment.getExternalStorageDirectory();
File dir = new File (root.getAbsolutePath() + "/download");
if (!(dir.exists())) {
dir.mkdirs();
}
Log.d("WriteSDCard", "Start writing");
File file = new File(dir, "myData.txt");
try {
FileOutputStream f = new FileOutputStream(file, true);
f.write(inputMsg);
f.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
Log.i(TAG, "******* File not found. Did you" +
" add a WRITE_EXTERNAL_STORAGE permission to the manifest?");
} catch (IOException e) {
e.printStackTrace();
}
}
read() returns the number of bytes that were read, or -1. You are ignoring both possibilities, and assuming that it filled the buffer. All you have to do is store the result in a variable, check for -1, and otherwise pass it to the write() method.
Actually you should pass the input stream to your method, and use a loop after creating the file:
int count;
byte[] buffer = new byte[8192];
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Your statement in a now-deleted comment that a new input stream is created per packet is not correct.
Here is my source code. I got the reading part but need a simple logic for my writing part which I"m not getting. Here in my current logic, data gets overwritten and I'm always able to see the last block of read data in my written file.
import java.io.File;
import java.io.FileInputStream;
java.io.FileNotFoundException;
java.io.FileOutputStream;
import java.io.IOException;
public class LoadTest
{
public void readFiles(File file) throws FileNotFoundException
{
int fsize = (int) file.length();
int part = (fsize/4)+(fsize%4);
byte[] block = new byte[part];
FileInputStream fin = new FileInputStream(file);
try
{
int val=-1;
do
{
int bytesread =0;
while(bytesread<part)
{
val = fin.read(block, bytesread, part-bytesread);
if (val<0)
break;
bytesread += val;
}
writeFiles(block,bytesread);
}
while(val>=0);
fin.close();
}
catch(IOException e)
{
e.printStackTrace();
}
}
public void writeFiles(byte[] block, int len) throws IOException
{
int byteswritten = 0;
FileOutputStream fout = new FileOutputStream("input.csv");
fout.write(block, byteswritten, len+byteswritten);
byteswritten +=len;
fout.close();
}
public static void main(String[] args) throws FileNotFoundException
{
LoadTest testobj = new LoadTest();
String folder = "/Users/NiranjanSubramanian/Desktop/TestFiles";
File dir = new File(folder);
File[] files = dir.listFiles();
System.out.println("Started");
for( File entry: files)
{
testobj.readFiles(entry);
}
System.out.println("Ended");
}
}
See my comments for how to solve the issue in a simple manner. However, let me suggest to you a simple alternative to do what you're asking.
final Path dir = Paths.get("/Users/NiranjanSubramanian/Desktop/TestFiles");
try (final OutputStream out = Files.newOutputStream(Paths.get("input.csv"))) {
for (final Path file : Files.newDirectoryStream(dir)) {
Files.copy(file, out);
}
}
This relies on Java 7's new file API but is (at least in my opinion) a far cleaner solution.
There are some libs that handle it for you, or even NIO, but the simplest way to do it is the following:
int read = 0;
byte[] buff = new byte[1024];
FileInputStream fis = new FileInputStream(yourInputFile);
FileOutputStream fos = new FileOutputStream(yourOutputFile);
while((read = fis.read(buff)) >= 0){
fos.write(buff, 0, read);
}
fos.flush();
fos.close();
fis.close();
Open the file in append mode.. your code will override since the default is overwrite, not append.
To append you need to pass the append parameter as true.
change
FileOutputStream fout = new FileOutputStream("input.csv");
to
FileOutputStream fout = new FileOutputStream("input.csv", true);
How to copy streams in Java:
byte[] buffer = new byte[8192]; // or whatever you like really
int count;
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Note that you don't need the entire input in memory before you start writing, so you are saving both time and space; and handling partial reads including the probable final one is trivially easy.
I am creating a program that will extract a zip and then insert the files into a database, every so often I get the error
java.lang.Exception: java.io.EOFException: Unexpected end of ZLIB input stream
I can not pinpoint the reason for this as the extraction code is pretty much the same as all the other code you can find on the web. My code is as follows:
public void extract(String zipName, InputStream content) throws Exception {
int BUFFER = 2048;
//create the zipinputstream
ZipInputStream zis = new ZipInputStream(content);
//Get the name of the zip
String containerName = zipName;
//container for the zip entry
ZipEntry entry;
// Process each entry
while ((entry = zis.getNextEntry()) != null) {
//get the entry file name
String currentEntry = entry.getName();
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
// establish buffer for writing file
byte data[] = new byte[BUFFER];
int currentByte;
// read and write until last byte is encountered
while ((currentByte = zis.read(data, 0, BUFFER)) != -1) {
baos.write(data, 0, currentByte);
}
baos.flush(); //flush the buffer
//this method inserts the file into the database
insertZipEntry(baos.toByteArray());
baos.close();
}
catch (Exception e) {
System.out.println("ERROR WITHIN ZIP " + containerName);
}
}
}
This is probably caused by this JVM bug (JVM-6519463)
I previously has about one or two errors on 1000 randomly created documents, I applied the proposed solution (catch EOFException and do nothing with it) and I have no more errors.
I would say you are occasionally being given truncated Zip files to process. Check upstream.
I had the same exception and the problem was in the compressing method (not extracting). I did not close the ZipOutputStream with zos.closeEntry() after writing to the output stream. Without that, compressing worked well but I got an exception while extracting.
public static byte[] zip(String outputFilename, byte[] output) {
try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
ZipOutputStream zos = new ZipOutputStream(baos)) {
zos.putNextEntry(new ZipEntry(outputFilename));
zos.write(output, 0, output.length);
zos.closeEntry(); //this line must be here
return baos.toByteArray();
} catch (IOException e) {
//catch exception
}
}
Never attempt to read more bytes than the entry contains. Call ZipEntry.getSize() to get the actual size of the entry, then use this value to keep track of the number of bytes remaining in the entry while reading from it. See below :
try{
...
int bytesLeft = (int)entry.getSize();
while ( bytesLeft>0 && (currentByte=zis.read(data, 0, Math.min(BUFFER, bytesLeft))) != -1) {
...
}
...
}
I'm trying to find a way to send files of different file types from a server to a client.
I have this code on the server to put the file into a byte array:
File file = new File(resourceLocation);
byte[] b = new byte[(int) file.length()];
FileInputStream fileInputStream;
try {
fileInputStream = new FileInputStream(file);
try {
fileInputStream.read(b);
} catch (IOException ex) {
System.out.println("Error, Can't read from file");
}
for (int i = 0; i < b.length; i++) {
fileData += (char)b[i];
}
}
catch (FileNotFoundException e) {
System.out.println("Error, File Not Found.");
}
I then send fileData as a string to the client. This works fine for txt files but when it comes to images I find that although it creates the file fine with the data in, the image won't open.
I'm not sure if I'm even going about this the right way.
Thanks for the help.
Don't put it into a string with a char cast. Just have your socket write the byte array you get from the file input stream.
If you're reading/writing binary data you should use byte streams (InputStream/OutputStream) instead of character streams and try to avoid conversions between bytes and chars like you did in your example.
You can use the following class to copy bytes from an InputStream to an OutputStream:
public class IoUtil {
private static final int bufferSize = 8192;
public static void copy(InputStream in, OutputStream out) throws IOException {
byte[] buffer = new byte[bufferSize];
int read;
while ((read = in.read(buffer, 0, bufferSize)) != -1) {
out.write(buffer, 0, read);
}
}
}
You don't give too much details of how you connect with the client. This is a minimal example showing how to stream some bytes to the client of a servlet. (You'll need to set some headers in the response and release the resources appropiately).
public class FileServlet extends HttpServlet {
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
// Some code before
FileInputStream in = new FileInputStream(resourceLocation);
ServletOutputStream out = response.getOutputStream();
IoUtil.copy(in, out);
// Some code after
}
}