We Keep Coding

Read large file error "outofmemoryerror"(java)

sorry for my english. I want to read a large file, but when I read error occurs outOfMemoryError. I do not understand how to work with memory in the application. The following code does not work:
try {
StringBuilder fileData = new StringBuilder(1000);
BufferedReader reader = new BufferedReader(new FileReader(file));
char[] buf = new char[8192];
int bytesread = 0,
bytesBuffered = 0;
while( (bytesread = reader.read( buf )) > -1 ) {
String readData = String.valueOf(buf, 0, bytesread);
bytesBuffered += bytesread;
fileData.append(readData); //this is error
if (bytesBuffered > 1024 * 1024) {
bytesBuffered = 0;
}
}
System.out.println(fileData.toString().toCharArray());
} finally {
}

You need pre allocate a large buffer to avoid reallocate.
File file = ...;
StringBuilder fileData = new StringBuilder(file.size());
And running with large heap size:
java -Xmx2G
==== update
A while loop using buffer doesn't need too memory to run. Treat input like a stream, match your search string with the stream. It's a really simple state machine. If you need search multiple words, you can find a TrieTree implementation(support stream) for that.
// the match state model
...xxxxxxabxxxxxaxxxxxabcdexxxx...
ab a abcd
File file = new File("path_to_your_file");
String yourSearchWord = "abcd";
int matchIndex = 0;
boolean matchPrefix = false;
try (BufferedReader reader = new BufferedReader(new FileReader(file))) {
int chr;
while ((chr = reader.read()) != -1) {
if (matchPrefix == false) {
char searchChar = yourSearchWord.charAt(0);
if (chr == searchChar) {
matchPrefix = true;
matchIndex = 0;
}
} else {
char searchChar = yourSearchWord.charAt(++matchIndex);
if (chr == searchChar) {
if (matchIndex == yourSearchWord.length() - 1) {
// match!!
System.out.println("match: " + matchIndex);
matchPrefix = false;
matchIndex = 0;
}
} else {
matchPrefix = false;
matchIndex = 0;
}
}
}
}

Try this. This might be helpful :-
try{
BufferedReader reader = new BufferedReader(new FileReader(file));
String txt = "";
while( (txt = reader.read()) != null){
System.out.println(txt);
}
}catch(Exception e){
System.out.println("Error : "+e.getMessage());
}

You should not hold such big files in memory, because you run out of it, as you see. Since you use Java 7, you need to read the file manually as stream and check the content on the fly. Otherwise you could use the stream API of Java 8. This is just an example. It works, but keep in mind, that the position of the found word could vary due to encoding issues, so this is no production code:
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
public class FileReader
{
private static String wordToFind = "SEARCHED_WORD";
private static File file = new File("YOUR_FILE");
private static int currentMatchingPosition;
private static int foundAtPosition = -1;
private static int charsRead;
public static void main(String[] args) throws IOException
{
try (FileInputStream fis = new FileInputStream(file))
{
System.out.println("Total size to read (in bytes) : " + fis.available());
int c;
while ((c = fis.read()) != -1)
{
charsRead++;
checkContent(c);
}
if (foundAtPosition > -1)
{
System.out.println("Found word at position: " + (foundAtPosition - wordToFind.length()));
}
else
{
System.out.println("Didnt't find the word!");
}
}
catch (IOException e)
{
e.printStackTrace();
}
}
private static void checkContent(int c)
{
if (currentMatchingPosition >= wordToFind.length())
{
//already found....
return;
}
if (wordToFind.charAt(currentMatchingPosition) == (char)c)
{
foundAtPosition = charsRead;
currentMatchingPosition++;
}
else
{
currentMatchingPosition = 0;
foundAtPosition = -1;
}
}
}

Why Is my code only sending part of a Large File

Im trying to make a Java Socket File Server but I have hit a dead end, It seems to be working until in around loop 4080 then seems to stop, Any ideas as to why this is happening? here is my code:
public class FileSender extends Thread{
private final int PORT = 7777;
private Socket sock;
private DataInputStream fileStream;
private OutputStream out;
private File file ;
public void run() {
try {
file = new File("path");
if(file.exists()){
System.out.println("Found File");
if(file.canRead()){
System.out.println("Can Read File");
}
}
this.fileStream = new DataInputStream(new FileInputStream(file));
sock = new Socket("localhost",PORT);
out = sock.getOutputStream();
copyStream(fileStream, out, file);
} catch (IOException ex) {
Logger.getLogger(FileSender.class.getName()).log(Level.SEVERE, null, ex);
}
}
public void copyStream(DataInputStream in, OutputStream out, File file) throws IOException{
byte[] buf = new byte[1024];
int total = 0;
while(file.getTotalSpace() != total){
int r = in.read(buf);
if(r != -1){
out.write(buf, 0, r);
}
total += r;
}
out.flush();
System.out.println("Total was:" + total);
}
}
This is My Server:
public class FileReceiver extends Thread {
private final int PORT = 7777;
private ServerSocket sSoc;
private DataInputStream in;
public void run() {
try {
byte[] buf = new byte[1024];
sSoc = new ServerSocket(PORT);
Socket conn = sSoc.accept();
in = new DataInputStream(new BufferedInputStream(conn.getInputStream()));
File file = new File("C:\\test.rar");
if (!file.exists()) {
file.createNewFile();
}
if (file.canWrite()) {
FileOutputStream fos = new FileOutputStream(file);
int x= 0 ;
do {
in.read(buf);
System.out.println(x + " : " + buf);
fos.write(buf);
x++;
} while (in.read(buf) != -1);
System.out.println("Complete");
}
} catch (IOException ex) {
Logger.getLogger(FileReceiver.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
EDIT : The program will send a small text file but onlky sends part of a Larger File.

Your condition in the while loop in copyStream seems wrong. Please try and change it to the following and try.
public void copyStream(DataInputStream in, OutputStream out, File file) throws IOException{
byte[] buf = new byte[1024];
int total = 0;
while(true){
int r = in.read(buf);
if(r != -1){
out.write(buf, 0, r);
total += r;
} else {
break;
}
}

The problem was in my server I was calling in.read() twice forcing anything larger then a single buffer size to miss segments.

Unable to save and retrieve using ByteAarrayOutputStream

I am making this J2ME application but I am having some problem when I am trying to save I thinks that it save properly but I am not sure....but when I retrieve it gives null
This is how I am storing them
PAR par = new PAR(oldMonPay, newMonPay, oldInterest);
par.setOldMPay(oldMonPay);
par.setNewMPay(newMonPay);
par.setOldInt(oldInterest);
And this is how I saving and retrieving
public static byte[] parseObjPAR(PAR p) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputStream out;
try {
out = new DataOutputStream(baos);
out.writeUTF(p.getNewMPay());
out.writeUTF(p.getOldInt());
out.writeUTF(p.getOldMPay());
} catch (IOException e) {
}
return baos.toByteArray();
}
public static PAR parseByteArrPAR(byte[] b) {
PAR p = null;
ByteArrayInputStream bais;
DataInputStream in;
if (b != null) {
try {
bais = new ByteArrayInputStream(b);
in = new DataInputStream(bais);
p = new PAR(
in.readUTF(),
in.readUTF(),
in.readUTF());
} catch (IOException e) {
}
}
return p;
}
This is how I displaying the retrieved information, there is another problem this thing is not showing all the data but is only showing the 3 records. I think the first 3.
public void populatePAResult(PAR[] p) {
try {
for (int i = 0; i < p.length; i++) {
String oldMP = p[i].getOldMPay();
String newMP = p[i].getNewMPay();
String oldI = p[i].getOldInt();
result1.append("Day : " + oldMP, null);
result1.append("Time : " + oldI, null);
result1.append("Technology : " + newMP, null);
}
} catch (Exception e) {
}
}

In the parseObjPAR method that writes the data the order is:
out.writeUTF(p.getNewMPay());
out.writeUTF(p.getOldInt());
out.writeUTF(p.getOldMPay());
whereas when you read it back in and pass the order the constructor is expecting is different:
PAR par = new PAR(oldMonPay, newMonPay, oldInterest);
so even if it wasn't null the loaded data would be invalid.

compress base64 encoded string data into decode format

Hi im decoding the base64 string data and saving into the database like url format with time stamp so i need to compress the decoded data and my compression like if the coming data is less than 100kb then no need to compress otherwise compress the data into 50% off.
try
{
String FileItemRefPath = propsFPCConfig.getProperty("fileCreationReferencePath");
String imageURLReferncePath = propsFPCConfig.getProperty("imageURLReferncePath");
File f = new File(FileItemRefPath+"/"+"productimages"+"/"+donorId);
String strException = "Actual File "+f.getName();
if(!f.exists())
{
if (!f.mkdirs())
{
System.out.println("direction creation failed");
return null;
}
}
else
{
boolean isdirCreationStatus = f.mkdirs();
}
String strDateTobeAppended = new SimpleDateFormat("yyyyMMddhhmmss").format(new Date());
String fileName = strImageName+strDateTobeAppended;
savedFile = new File(f.getAbsolutePath()+"/"+fileName);
strException=strException+" savedFile "+savedFile.getName();
Base64 decoder = new Base64();
byte[] decodedBytes = decoder.decode(strImageBase64);
if( (decodedBytes != null) && (decodedBytes.length != 0) )
{
System.out.println("Decoded bytes length:"+decodedBytes.length);
fos = new FileOutputStream(savedFile);
System.out.println(new String(decodedBytes) + "\n") ;
int x=0;
{
fos.write(decodedBytes, 0, decodedBytes.length);
}
fos.flush();
}
if(fos != null)
{
fos.close();
System.out.println("file output stream"+savedFile.getName());
return savedFile.getName();
}
else
{
return null;
}
}
catch(Exception e)
{
e.printStackTrace();
}
finally
{
try
{
if( fos!= null)
{
fos.close();
}
else
{
savedFile = null;
}
}
catch (IOException e)
{
e.printStackTrace();
}
}
return savedFile.getName();
}

Your main problem is probably not going to be compressing it in Java. For that you can simply use something like the Deflater class. On iOS however it might be a little more complicated, as I'm not sure what kind of zlib-like tools you have available in iOS.

How do I concatenate sequential files in order with Java?

I have a directory that contains sequentially numbered log files and some Excel spreadsheets used for analysis. The log file are ALWAYS sequentially numbered beginning at zero, but the number of them can vary. I am trying to concatenate the log files, in the order they were created into a single text file which will be a concatenation of all the log files.
For instance, with log files foo0.log, foo1.log, foo2.log would be output to concatenatedfoo.log by appending foo1 after foo0, and foo2 after foo1.
I need to count all the files in the given directory with the extension of *.log, using the count to drive a for-loop that also generates the file name for concatenation. I'm having a hard time finding a way to count the files using a filter...none of the Java Turtorials on file operations seem to fit the situation, but I'm sure I'm missing something. Does this approach make sense? or is there an easier way?
int numDocs = [number of *.log docs in directory];
//
for (int i = 0; i <= numberOfFiles; i++) {
fileNumber = Integer.toString(i);
try
{
FileInputStream inputStream = new FileInputStream("\\\\Path\\to\\file\\foo" + fileNumber + ".log");
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream));
try
{
BufferedWriter metadataOutputData = new BufferedWriter(new FileWriter("\\\\Path\\to\\file\\fooconcat.log").append());
metadataOutputData.close();
}
//
catch (IOException e) // catch IO exception writing final output
{
System.err.println("Exception: ");
System.out.println("Exception: "+ e.getMessage().getClass().getName());
e.printStackTrace();
}
catch (Exception e) // catch IO exception reading input file
{
System.err.println("Exception: ");
System.out.println("Exception: "+ e.getMessage().getClass().getName());
e.printStackTrace();
}
}

how about
public static void main(String[] args){
final int BUFFERSIZE = 1024 << 8;
File baseDir = new File("C:\\path\\logs\\");
// Get the simple names of the files ("foo.log" not "/path/logs/foo.log")
String[] fileNames = baseDir.list(new FilenameFilter() {
#Override
public boolean accept(File dir, String name) {
return name.endsWith(".log");
}
});
// Sort the names
Arrays.sort(fileNames);
// Create the output file
File output = new File(baseDir.getAbsolutePath() + File.separatorChar + "MERGED.log");
try{
BufferedOutputStream out = new BufferedOutputStream(new FileOutputStream(output), BUFFERSIZE);
byte[] bytes = new byte[BUFFERSIZE];
int bytesRead;
final byte[] newLine = "\n".getBytes(); // use to separate contents
for(String s : fileNames){
// get the full path to read from
String fullName = baseDir.getAbsolutePath() + File.separatorChar + s;
BufferedInputStream in = new BufferedInputStream(new FileInputStream(fullName),BUFFERSIZE);
while((bytesRead = in.read(bytes,0,bytes.length)) != -1){
out.write(bytes, 0, bytesRead);
}
// close input file and ignore any issue with closing it
try{in.close();}catch(IOException e){}
out.write(newLine); // seperation
}
out.close();
}catch(Exception e){
throw new RuntimeException(e);
}
}
This code DOES assume that the "sequential naming" would be zero padded such that they will lexigraphically (?? sp) sort correctly. i.e. The files would be
0001.log (or blah0001.log, or 0001blah.log etc)
0002.log
....
0010.log
and not
1.log
2.log
...
10.log
The latter pattern will not sort correctly with the code I have given.

Here's some code for you.
File dir = new File("C:/My Documents/logs");
File outputFile = new File("C:/My Documents/concatenated.log");
Find the ".log" files:
File[] files = dir.listFiles(new FilenameFilter() {
#Override
public boolean accept(File file, String name) {
return name.endsWith(".log") && file.isFile();
}
});
Sort them into the appropriate order:
Arrays.sort(files, new Comparator<File>() {
#Override
public int compare(File file1, File file2) {
return numberOf(file1).compareTo(numberOf(file2));
}
private Integer numberOf(File file) {
return Integer.parseInt(file.getName().replaceAll("[^0-9]", ""));
}
});
Concatenate them:
byte[] buffer = new byte[8192];
OutputStream out = new BufferedOutputStream(new FileOutputStream(outputFile));
try {
for (File file : files) {
InputStream in = new FileInputStream(file);
try {
int charCount;
while ((charCount = in.read(buffer)) >= 0) {
out.write(buffer, 0, charCount);
}
} finally {
in.close();
}
}
} finally {
out.flush();
out.close();
}

By having the log folder as a File object, you can code like this
for (File logFile : logFolder.listFiles()){
if (logFile.getAbsolutePath().endsWith(".log")){
numDocs++;
}
}
to find the number of log files.

I would;
open the output file once. Just use a PrintWriter.
in a loop ...
create a File for each possible file
if it doesn't exist break the loop.
Using a BufferedReader
to read the lines of the file with readLine()
write each line to the output file.
You should be able to do this with about 12 lines of code. I would pass the IOExceptions to the caller.

You can use SequenceInputStream for concatenation of FileInputStreams.
To see all log files File.listFiles(FileFilter) can be used.
It will give you unsorted array with files. To sort files in right order, use Arrays.sort.
Code example:
static File[] logs(String dir) {
File root = new File(dir);
return root.listFiles(new FileFilter() {
#Override
public boolean accept(File pathname) {
return pathname.isFile() && pathname.getName().endsWith(".log");
}
});
}
static String cat(final File[] files) throws IOException {
Enumeration<InputStream> e = new Enumeration<InputStream>() {
int index;
#Override
public boolean hasMoreElements() {
return index < files.length;
}
#Override
public InputStream nextElement() {
index++;
try {
return new FileInputStream(files[index - 1]);
} catch (FileNotFoundException ex) {
throw new RuntimeException("File not available!", ex);
}
}
};
SequenceInputStream input = new SequenceInputStream(e);
StringBuilder sb = new StringBuilder();
int c;
while ((c = input.read()) != -1) {
sb.append((char) c);
}
return sb.toString();
}
public static void main(String[] args) throws IOException {
String dir = "<path-to-dir-with-logs>";
File[] logs = logs(dir);
for (File f : logs) {
System.out.println(f.getAbsolutePath());
}
System.out.println();
System.out.println(cat(logs));
}