I have a text file with a sequence of 4194304 letters ranging from A-D all on one line (4 MB).
How would I randomly point to a character and replace the following set of characters to another file that is 100 characters long and write it out to a file?
I'm actually currently able to do this, but I feel it's really inefficient when I iterate it several times.
Here's an illustration of what I mentioned above:
Link to Imageshack
Here's how I'm currently achieving this:
Random rnum = new Random();
FileInputStream fin = null;
FileOutputStream fout = null;
int count = 10000;
FileInputStream fin1 = null;
File file1 = new File("fileWithSet100C.txt");
int randChar = 0;
while(cnt > 0){
try {
int c = 4194304 - 100;
randChar = rnum.nextInt(c);
File file = new File("file.txt");
//seems inefficient to initiate these guys over and over
fin = new FileInputStream(file);
fin1 = new FileInputStream(file1);
//would like to remove this and have it just replace the original
fout = new FileOutputStream("newfile.txt");
int byte_read;
int byte_read2;
byte[] buffer = new byte[randChar];
byte[] buffer2 = new byte[(int)file1.length()]; //4m
byte_read = fin.read(buffer);
byte_read2 = fin1.read(buffer2);
fout.write(buffer, 0, byte_read);
fout.write(buffer2, 0, byte_read2);
byte_read = fin.read(buffer2);
buffer = new byte[4096]; //4m
while((byte_read = (fin.read(buffer))) != -1){
fout.write(buffer, 0, byte_read);
}
cnt--;
}
catch (...) {
...
}
finally {
...
}
try{
File file = new File("newfile.txt");
fin = new FileInputStream(file);
fout = new FileOutputStream("file.txt");
int byte_read;
byte[] buffer = new byte[4096]; //4m
byte_read = fin.read(buffer);
while((byte_read = (fin.read(buffer))) != -1){
fout.write(buffer, 0, byte_read);
}
}
catch (...) {
...
}
finally {
...
}
Thanks for reading!
EDIT:
For those curious, here's the code I used to solve the aforementioned problem:
String stringToInsert = "insertSTringHERE";
byte[] answerByteArray = stringToInsert.getBytes();
ByteBuffer byteBuffer = ByteBuffer.wrap(answerByteArray);
Random rnum = new Random();
randChar = rnum.nextInt(4194002); //4MB worth of bytes
File fi = new File("file.txt");
RandomAccessFile raf = null;
try {
raf = new RandomAccessFile(fi, "rw");
} catch (FileNotFoundException e1) {
// TODO error handling and logging
}
FileChannel fo = null;
fo = raf.getChannel();
// Move to the beginning of the file and write out the contents
// of the byteBuffer.
try {
outputFileChannel.position(randChar);
while(byteBuffer.hasRemaining()) {
fo.write(byteBuffer);
}
} catch (IOException e) {
// TODO error handling and logging
}
try {
outputFileChannel.close();
} catch (IOException e) {
// TODO error handling and logging
}
try {
randomAccessFile.close();
} catch (IOException e) {
// TODO error handling and logging
}
You probably want to use Java's random-access file features. Sun/Oracle has a Random Access Files tutorial that will probably be useful to you.
If you can't use Java 7, then look at RandomAccessFile which also has seek functionality and has existed since Java 1.0.
First off, for your files you could have the Files as global variables. This would all you to use the file when ever you needed without reading it again. Also note that if you keep making new files then you will lose the data that you have already acquired.
For example:
public class Foo {
// Gloabal Vars //
File file;
public Foo(String location) {
// Do Something
file = new File(location);
}
public add() {
// Add
}
}
Answering your question, I would first read both files and then make all the changes you want in memory. After you have made all the changes, I would then write the changes to the file.
However, if the files are very large, then I would make all the changes one by one on the disk... it will be slower, but you will not run out of memory this way. For what you are doing I doubt you could use a buffer to help counter how slow it would be.
My overall suggestion would be to use arrays. For example I would do the following...
public char[] addCharsToString(String str, char[] newChars, int index) {
char[] string = str.toCharArray();
char[] tmp = new char[string.length + newChars.length];
System.arraycopy(string, 0, tmp, 0, index);
System.arraycopy(newChars, index, tmp, index, newChars.length);
System.arraycopy(string, index + newChars.length, tmp, index + newChars.length, tmp.length - (newChars.length + index));
return tmp;
}
Hope this helps!
Related
I am trying to transfer a .mp4 file using WebRTC and it's DataChannel. In order to do that I am breaking the file into chunks like below:
FileInputStream is = new FileInputStream(file);
byte[] chunk = new byte[260000];
int chunkLen = 0;
sentFileByte = new ArrayList<>();
while ((chunkLen = is.read(chunk)) != -1) {
sentFileByte.add(chunk);
}
After that, sending the chunks by index like:
byte[] b = sentFileByte.get(index);
ByteBuffer bb = ByteBuffer.wrap(b);
bb.put(b);
bb.flip();
dataChannel.send(new DataChannel.Buffer(bb, true));
On the receiver end I am receiving the chunks and adding it to an Arraylist
receivedFileByteArr.add(chunkByteArr);
After receiving all the chunks successfully I am trying to convert these in to a file like below:
String path = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS).getAbsolutePath() + "/" + fileName;
File file = new File(path);
try {
FileOutputStream fileOutputStream = new FileOutputStream(file);
for (int i = 0; i < receivedFileByteArr.size(); i++) {
fileOutputStream.write(receivedFileByteArr.get(i));
}
fileOutputStream.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
After completing all these steps, file is created successfully. File size is also same. But problem is the file is not playable in any video player. I guess I am making some mistake on FileInputStream and FileOutputStream. I need help to fix this error.
I've made a basic client server FTP program using sockets, but for some reason files are getting corrupted during the transfer. In the case below, I'm pushing a file to the server from the client. It almost works, since some files (such as a .png) transfer and open fine, but others (a .docx) don't. Any file that I transfer has a different MD5 to the one I sent.
Client code:
File file = null;
FTPDataBlock transferBlock;
int numBytesRead = 0;
int blockNumber = 1;
int blockSize = 1024;
byte[] block = new byte[blockSize];
fc = new JFileChooser();
// select file to upload
int returnVal = fc.showOpenDialog(Client.this);
if (returnVal == JFileChooser.APPROVE_OPTION) {
file = fc.getSelectedFile();
try {
// get total number of blocks and send to server
int totalNumBlocks = (int)Math.ceil((file.length()*1.0) / blockSize);
System.out.println("File length is: " + file.length());
FTPCommand c = new FTPCommand("PUSH", Integer.toString(totalNumBlocks));
oos = new ObjectOutputStream(sock.getOutputStream());
oos.writeObject(c);
oos.flush();
// send to server block by block
FileInputStream fin = new FileInputStream(file);
while ((numBytesRead = fin.read(block)) != -1){
transferBlock = new FTPDataBlock(file.getName(), blockNumber, block);
blockNumber++;
System.out.println("Sending block " + transferBlock.getBlockNumber() + " of " + totalNumBlocks);
oos = new ObjectOutputStream(sock.getOutputStream());
oos.writeObject(transferBlock);
oos.flush();
}
fin.close();
System.out.println("PUSH Complete");
// get response from server
ois = new ObjectInputStream(sock.getInputStream());
FTPResponse response = (FTPResponse)ois.readObject();
statusArea.setText(response.getResponse());
} catch (IOException | ClassNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
}
Server Code:
else if (cmd.getCommand().equals("PUSH")){
// get total number of file blocks
int totalNumBlocks = Integer.parseInt(cmd.getParameters());
// get first block
in = new ObjectInputStream(sock.getInputStream());
FTPDataBlock currentBlock = (FTPDataBlock)in.readObject();
// create file and write first block to file
File file = new File (workingDirectory + File.separator + currentBlock.getFilename());
FileOutputStream fOut = new FileOutputStream(file);
fOut.write(currentBlock.getData());
fOut.flush();
// get remaining blocks
while(currentBlock.getBlockNumber()+1 <= totalNumBlocks){
in = new ObjectInputStream(sock.getInputStream());
currentBlock = (FTPDataBlock)in.readObject();
fOut.write(currentBlock.getData());
fOut.flush();
}
fOut.close();
// send response
FTPResponse response = new FTPResponse("File Received OK");
out = new ObjectOutputStream(sock.getOutputStream());
out.writeObject(response);
}
FTPDataBlock class:
public class FTPDataBlock implements Serializable{
private static final long serialVersionUID = 1L;
private String filename;
private int blockNumber; // current block number
private byte[] data;
//constructors & accessors
}
I'm sure it's something small that I'm missing here. Any ideas?
This happened because the server was writing whole 1024 byte blocks to the file, even if there was less than 1024 bytes actually written to the block.
The solution (thanks to #kdgregory) was to use the return value of FileInputStream.read() to populate a new attribute in my FTPDataBlock class, int bytesWritten.
Then on the server side I could use:
FileOutputStream.write(currentBlock.getData(), 0, currentBlock.getBytesWritten());
to write the exact number of bytes to the file, instead of the whole block every time.
I think there may be a problem with the file extension. provide a option in the client side as:
FILE_TO_RECEIVED = JOptionPane.showInputDialog("Please enter the Drive followed by the file name to be saved. Eg: D:/xyz.jpg");
it is to help you to provide the correct file extension name.
then i think u should also provide the file size in client side like:
public final static int FILE_SIZE = 6022386;
and in then in the array block u used u can make the following changes as:
try {
sock = new Socket(SERVER, SOCKET_PORT);
byte [] mybytearray = new byte [FILE_SIZE];
InputStream is = sock.getInputStream();
fos = new FileOutputStream(FILE_TO_RECEIVED);
bos = new BufferedOutputStream(fos);
bytesRead = is.read(mybytearray,0,mybytearray.length);
current = bytesRead;
do {
bytesRead =
is.read(mybytearray, current, (mybytearray.length-current));
if(bytesRead >= 0) current += bytesRead;
} while(bytesRead > -1);
bos.write(mybytearray, 0 , current);
bos.flush();
}
I'm trying to send large files via socket. The program works fine for small files (such as html pages or pdf), but when i send files over 3/4 mb the output is always corrupted (viewing it with a text editor i noticed that the last few lines are always missing).
Here's the code of the server:
BufferedInputStream in = null;
FileOutputStream fout = null;
try {
server = new ServerSocket(port);
sock = server.accept();
in = new BufferedInputStream(sock.getInputStream());
setPerc(0);
received = 0;
int incByte = -1;
fout = new FileOutputStream(path+name, true);
long size = length;
do{
int buffSize;
if(size >= 4096){
buffSize = 4096;
}else{
buffSize = 1;
}
byte[] o = new byte[buffSize];
incByte = in.read(o, 0, buffSize);
fout.write(o);
received+=buffSize;
setPerc(calcPerc(received, length));
size -= buffSize;
//d("BYTE LETTI => "+incByte);
}while(size > 0);
server.close();
} catch (IOException e) {
e("Errore nella ricezione file: "+e);
}finally{
try {
fout.flush();
fout.close();
in.close();
} catch (IOException e) {
e("ERRORE INCOMINGFILE");
}
}
pr.release(port);
And here's the code of the client:
FileInputStream fin = null;
BufferedOutputStream out = null;
try {
sock = new Socket(host, port);
fin = new FileInputStream(file);
out = new BufferedOutputStream(sock.getOutputStream());
long size = file.length();
int read = -1;
do{
int buffSize = 0;
if(size >= 4096){
buffSize = 4096;
}else{
buffSize = (int)size;
}
byte[] o = new byte[buffSize];
for(int i = 0; i<o.length;i++){
o[i] = (byte)0;
}
read = fin.read(o, 0, buffSize);
out.write(o);
size -= buffSize;
//d("BYTE LETTI DAL FILE => "+read);
}while(size > 0);
} catch (UnknownHostException e) {
} catch (IOException e) {
d("ERRORE NELL'INVIO DEL FILE: "+e);
e.printStackTrace();
}finally{
try {
out.flush();
out.close();
fin.close();
} catch (IOException e) {
d("Errore nella chiusura dei socket invio");
}
}
i think it's something related with the buffer size, but i can't figure out what's wrong here.
This is incorrect:
byte[] o = new byte[buffSize];
incByte = in.read(o, 0, buffSize);
fout.write(o);
You are reading up to buffSize bytes and then writing exactly buffSize bytes.
You are doing the same thing at the other end as well.
You may be able to get away with this when reading from a file1, but when you read from a socket then a read is liable to give you a partially filled buffer, especially if the writing end can't always keep ahead of the reading end 'cos you are hammering the network with a large transfer.
The right way to do it is:
incByte = in.read(o, 0, buffSize);
fout.write(o, 0, incByte);
1 - It has been observed that when you read from a local file, a read call will typically give you all of the bytes that you requested (subject to the file size, etc). So, if you set buffSize to the length of the file, this code would probably work when reading from a local file. But doing this is a bad idea, because you are relying behaviour that is not guaranteed by either Java or a typical operating system.
You might have a problem e.g. here.
read = fin.read(o, 0, buffSize);
out.write(o);
Here read gives you the count of bytes you've actually just read.
On the next line you should write out only as many bytes as you've read.
In other words, you cannot expect the size of the file
you're reading to be multiple of your buffer size.
Review your server code too for the same issue.
The correct way to copy streams in Java is as follows:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
where count is an int, and buffer is a byte[] array of length > 0, typically 8k. You don't need to allocate byte arrays inside the loop, and you don't need a byte array of a specific size. Specifically, it's a complete waste of space to allocate a buffer as large as the file; it only works up to files of Integer.MAX_VALUE bytes, and it doesn't scale.
You do need to save the count returned by 'read()' and use it in the 'write()' method as shown above.
I'm using Apache Commons Compress to create tar archives and decompress them. My problems start with this method:
private void decompressFile(File file) throws IOException {
logger.info("Decompressing " + file.getName());
BufferedOutputStream outputStream = null;
TarArchiveInputStream tarInputStream = null;
try {
tarInputStream = new TarArchiveInputStream(
new FileInputStream(file));
TarArchiveEntry entry;
while ((entry = tarInputStream.getNextTarEntry()) != null) {
if (!entry.isDirectory()) {
File compressedFile = entry.getFile();
File tempFile = File.createTempFile(
compressedFile.getName(), "");
byte[] buffer = new byte[BUFFER_MAX_SIZE];
outputStream = new BufferedOutputStream(
new FileOutputStream(tempFile), BUFFER_MAX_SIZE);
int count = 0;
while ((count = tarInputStream.read(buffer, 0, BUFFER_MAX_SIZE)) != -1) {
outputStream.write(buffer, 0, count);
}
}
deleteFile(file);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
if (outputStream != null) {
outputStream.flush();
outputStream.close();
}
}
}
Every time I run the code, compressedFile variable is null, but the while loop is iterating over all entries in my test tar.
Could you help me to understand what I'm doing wrong?
From the official documentation
Reading entries from an tar archive:
TarArchiveEntry entry = tarInput.getNextTarEntry();
byte[] content = new byte[entry.getSize()];
LOOP UNTIL entry.getSize() HAS BEEN READ {
tarInput.read(content, offset, content.length - offset);
}
I have written an example starting from your implementation and testing with a very trivial .tar (just one entry of text).
Not knowing the exact requirement I just take care of solving the problem of reading the archive avoiding the nullpointer. Debugging, the entry is available as you also have found
private static void decompressFile(File file) throws IOException {
BufferedOutputStream outputStream = null;
TarArchiveInputStream tarInputStream = null;
try {
tarInputStream = new TarArchiveInputStream(
new FileInputStream(file));
TarArchiveEntry entry;
while ((entry = tarInputStream.getNextTarEntry()) != null) {
if (!entry.isDirectory()) {
File compressedFile = entry.getFile();
String name = entry.getName();
int size = 0;
int c;
while (size < entry.getSize()) {
c = tarInputStream.read();
System.out.print((char) c);
size++;
}
(.......)
AS I said: I tested with a tar including only an entry of text (you can also try this approach to verify the code) to be sure that the null is avoided.
You need to make all the needed adaptations for your real needs.
It is clear that you will have to handle streams as in the metacode I posted on top.
It shows how to deal with the single entries.
Try using getNextEntry() method instead of getNextTarEntry() method.
The second method returns a TarArchiveEntry. Probably this is not what you want!
Here is my source code. I got the reading part but need a simple logic for my writing part which I"m not getting. Here in my current logic, data gets overwritten and I'm always able to see the last block of read data in my written file.
import java.io.File;
import java.io.FileInputStream;
java.io.FileNotFoundException;
java.io.FileOutputStream;
import java.io.IOException;
public class LoadTest
{
public void readFiles(File file) throws FileNotFoundException
{
int fsize = (int) file.length();
int part = (fsize/4)+(fsize%4);
byte[] block = new byte[part];
FileInputStream fin = new FileInputStream(file);
try
{
int val=-1;
do
{
int bytesread =0;
while(bytesread<part)
{
val = fin.read(block, bytesread, part-bytesread);
if (val<0)
break;
bytesread += val;
}
writeFiles(block,bytesread);
}
while(val>=0);
fin.close();
}
catch(IOException e)
{
e.printStackTrace();
}
}
public void writeFiles(byte[] block, int len) throws IOException
{
int byteswritten = 0;
FileOutputStream fout = new FileOutputStream("input.csv");
fout.write(block, byteswritten, len+byteswritten);
byteswritten +=len;
fout.close();
}
public static void main(String[] args) throws FileNotFoundException
{
LoadTest testobj = new LoadTest();
String folder = "/Users/NiranjanSubramanian/Desktop/TestFiles";
File dir = new File(folder);
File[] files = dir.listFiles();
System.out.println("Started");
for( File entry: files)
{
testobj.readFiles(entry);
}
System.out.println("Ended");
}
}
See my comments for how to solve the issue in a simple manner. However, let me suggest to you a simple alternative to do what you're asking.
final Path dir = Paths.get("/Users/NiranjanSubramanian/Desktop/TestFiles");
try (final OutputStream out = Files.newOutputStream(Paths.get("input.csv"))) {
for (final Path file : Files.newDirectoryStream(dir)) {
Files.copy(file, out);
}
}
This relies on Java 7's new file API but is (at least in my opinion) a far cleaner solution.
There are some libs that handle it for you, or even NIO, but the simplest way to do it is the following:
int read = 0;
byte[] buff = new byte[1024];
FileInputStream fis = new FileInputStream(yourInputFile);
FileOutputStream fos = new FileOutputStream(yourOutputFile);
while((read = fis.read(buff)) >= 0){
fos.write(buff, 0, read);
}
fos.flush();
fos.close();
fis.close();
Open the file in append mode.. your code will override since the default is overwrite, not append.
To append you need to pass the append parameter as true.
change
FileOutputStream fout = new FileOutputStream("input.csv");
to
FileOutputStream fout = new FileOutputStream("input.csv", true);
How to copy streams in Java:
byte[] buffer = new byte[8192]; // or whatever you like really
int count;
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Note that you don't need the entire input in memory before you start writing, so you are saving both time and space; and handling partial reads including the probable final one is trivially easy.