We Keep Coding

breaking the lines while chunking into multiple files

I am dividing my file into chunks but only problem i am facing is,
i have .srt file, but while doing chunks, it's cutting the characters i.e in first .srt file it's like 00:26:20,230 --> . in next file it continuing the next time stamp 00:27:40,343.
I need to check the timestamp to be complete and then next full subtitle sentence too. i.e if it's cutting the subtitle timesstamp or dialogue in in file, that tect to be append to next file. Please suggest me how can i achieve.
I am trying like below,
String FilePath = "/Users/meh/Desktop/escapeplan.srt";
FileInputStream fin = new FileInputStream(FilePath);
System.out.println("size: " +fin.getChannel().size());
long abc = 0l;
abc = (fin.getChannel().size())/3;
System.out.println("6: " +abc);
System.out.println("abc: " +abc);
//FilePath = args[1];
File filename = new File(FilePath);
long splitFileSize = 0,bytefileSize=0;
if (filename.exists()) {
try {
//bytefileSize = Long.parseLong(args[2]);
splitFileSize = abc;
Splitme spObj = new Splitme();
spObj.split(FilePath, (long) splitFileSize);
spObj = null;
} catch (Exception e) {
e.printStackTrace();
}
} else {
System.out.println("File Not Found....");
}
public void split(String FilePath, long splitlen) {
long leninfile = 0, leng = 0;
int count = 1, data;
try {
File filename = new File(FilePath);
InputStream infile = new BufferedInputStream(new FileInputStream(filename));
data = infile.read();
System.out.println("data");
System.out.println(data);
while (data != -1) {
filename = new File("/Users/meh/Documents/srt" + count + ".srt");
//RandomAccessFile outfile = new RandomAccessFile(filename, "rw");
OutputStream outfile = new BufferedOutputStream(new FileOutputStream(filename));
while (data != -1 && leng < splitlen) {
outfile.write(data);
leng++;
data = infile.read();
}
leninfile += leng;
leng = 0;
outfile.close();
changeTimeStamp(filename, count);
count++;
}
} catch (Exception e) {
e.printStackTrace();
}
}
i am trying to check the time stamp is in correct format or not. Then i need to check next line to be a dialogue and then the next line to be empty line. then it can stop chunk or else it should append the text from the previous chunk to next chunk file in the beginning of line . so that it may get in correct format.
I tried checking the format like,
while ((strLine = br.readLine()) != null) {
String[] atoms = strLine.split(" --> ");
if (atoms.length == 1) {
out.write(strLine + "\n");
} else {
String startTS = atoms[0];
String endTS = atoms[1];
System.out.print("sri atmos start" + startTS);
System.out.print("sri atmos end" + endTS);
SimpleDateFormat sdf = new SimpleDateFormat("HH:mm:ss,SSS");
sdf.setLenient(false);
try
{
sdf.parse(startTS);
sdf.parse(endTS);
System.out.println("Valid time");
System.out.println("File path" + srcFileNm);
}
catch(Exception e) {
System.out.println("Invalid time");
System.out.println("Exception start" + startTS);
System.out.println("Exception end" + endTS);
}
}
some screens of my output chunks,
Help me how can i make this possible.

I think you should change approach, and fully use basic I/O methods. I tried to encapsulate logic in a small class, that produces a triple with id, msecs and a list of subtitles (if I'm not wrong, you can have more than a line). Then I leaved the remainder externally. Chunker is a class that reads a triple (class Three) from file, so that you can manage it and write it somewhere.
This is just a "quick&dirty" idea that you can refine, but it should work.
package org.norsam.stackoverflow;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
public class Chunker
{
BufferedReader r;
int chunk = 0;
File dir;
public Chunker(File dir, String filename) throws IOException
{
File f = new File(dir, filename);
this.dir = dir;
this.r = new BufferedReader(new FileReader(f));
}
public Three readThree() throws IOException
{
Integer id = Integer.parseInt(r.readLine());
String msecs = r.readLine();
String s = null;
List<String> srt = new ArrayList<>();
while (!(s = r.readLine().trim()).isEmpty()) {
srt.add(s);
}
return new Three(id, msecs, srt);
}
class Three
{
Integer id;
String msecs;
List<String> srts;
Three(Integer id, String msecs, List<String> srts)
{
this.id = id;
this.msecs = msecs;
this.srts = srts;
}
Three doSomething() {
// here you can do something with your data,
// e.g. split msecs on "-->" and check times
return this;
}
void write(BufferedWriter r) throws IOException
{
r.write(id);
r.newLine();
r.write(msecs);
r.newLine();
for (String s : srts) {
r.write(s);
r.newLine();
}
r.newLine();
}
}
public static void main(String[] args) throws IOException
{
String baseDir = "/dir/where/resides/srt";
String filename = "filename.srt";
int elemPerChunk = 50;
int fileNum = 0;
File dir = new File(baseDir);
Chunker chunker = new Chunker(dir, filename);
boolean completed = false;
while (!completed) {
int srtCount = 0;
File f = new File(baseDir, "ch." + (fileNum++) + "." + filename);
BufferedWriter w = new BufferedWriter(new FileWriter(f));
try {
while (srtCount++ < elemPerChunk) {
chunker.readThree().doSomething().write(w);
}
} catch (NullPointerException e) {
completed = true;
}
w.close();
}
}
}

Read the newly appended file content to an InputStream in Java

I have a writer program that writes a huge serialized java object (at the scale of 1GB) into a binary file on local disk at a specific speed. Actually, the writer program (implemented in C language) is a network receiver that receives the bytes of the serialized object from a remote server. The implementation of the writer is fixed.
Now, I want to implement a Java reader program that reads the file and deserializes it to a Java object. Since the file could be very large, it is beneficial to reduce the latency of deserializing the object. Particularly, I want the Java reader starts to read/deserialize the object once the first byte of the object has been written to the disk file so that the reader can start to deserialize the object even before the entire serialized object has been written to the file. The reader knows the size of the file ahead of time (before the first byte is written to the file).
I think what I need is something like a blocking file InputStream that will be blocked when it reaches the EndOfFile but it has not read the expected number of bytes (the size of the file will be). Thus, whenever new bytes have been written to the file, the reader's InputStream could keep reading the new content. However, FileInputStream in Java does not support this feature.
Probably, I also need a file listener that monitoring the changes made to the file to achieve this feature.
I am wondering if there is any existing solution/library/package can achieve this function. Probably the question may be similar to some of the questions in monitoring the log files.
The flow of the bytes is like this:
FileInputStream -> SequenceInputStream -> BufferedInputStream -> JavaSerializer

You need two threads: Thread1 to download from the server and write to a File, and Thread2 to read the File as it becomes available.
Both threads should share a single RandomAccessFile, so access to the OS file can be synchronized correctly. You could use a wrapper class like this:
public class ReadWriteFile {
ReadWriteFile(File f, long size) throws IOException {
_raf = new RandomAccessFile(f, "rw");
_size = size;
_writer = new OutputStream() {
#Override
public void write(int b) throws IOException {
write(new byte[] {
(byte)b
});
}
#Override
public void write(byte[] b, int off, int len) throws IOException {
if (len < 0)
throw new IllegalArgumentException();
synchronized (_raf) {
_raf.seek(_nw);
_raf.write(b, off, len);
_nw += len;
_raf.notify();
}
}
};
}
void close() throws IOException {
_raf.close();
}
InputStream reader() {
return new InputStream() {
#Override
public int read() throws IOException {
if (_pos >= _size)
return -1;
byte[] b = new byte[1];
if (read(b, 0, 1) != 1)
throw new IOException();
return b[0] & 255;
}
#Override
public int read(byte[] buff, int off, int len) throws IOException {
synchronized (_raf) {
while (true) {
if (_pos >= _size)
return -1;
if (_pos >= _nw) {
try {
_raf.wait();
continue;
} catch (InterruptedException ex) {
throw new IOException(ex);
}
}
_raf.seek(_pos);
len = (int)Math.min(len, _nw - _pos);
int nr = _raf.read(buff, off, len);
_pos += Math.max(0, nr);
return nr;
}
}
}
private long _pos;
};
}
OutputStream writer() {
return _writer;
}
private final RandomAccessFile _raf;
private final long _size;
private final OutputStream _writer;
private long _nw;
}
The following code shows how to use ReadWriteFile from two threads:
public static void main(String[] args) throws Exception {
File f = new File("test.bin");
final long size = 1024;
final ReadWriteFile rwf = new ReadWriteFile(f, size);
Thread t1 = new Thread("Writer") {
public void run() {
try {
OutputStream w = new BufferedOutputStream(rwf.writer(), 16);
for (int i = 0; i < size; i++) {
w.write(i);
sleep(1);
}
System.out.println("Write done");
w.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
};
Thread t2 = new Thread("Reader") {
public void run() {
try {
InputStream r = new BufferedInputStream(rwf.reader(), 13);
for (int i = 0; i < size; i++) {
int b = r.read();
assert (b == (i & 255));
}
int eof = r.read();
assert (eof == -1);
r.close();
System.out.println("Read done");
} catch (IOException ex) {
ex.printStackTrace();
}
}
};
t1.start();
t2.start();
t1.join();
t2.join();
rwf.close();
}

PrintWriter doesn't save all characters

class Start
{
File plikIN;
Scanner in;
PrintWriter out;
Start(String input, String output)
{
plikIN = new File(input);
try{
in = new Scanner(plikIN);
out = new PrintWriter(output);
} catch (FileNotFoundException e) {System.out.println("Nie odnaleziono podanego pliku\n"+e);}
}
private void saveing() throws IOException
{
String word;
int wordLength;
String wordTable[];
char c;
while((word = in.next()) != null)
{
wordLength = word.length();
wordTable = new String[wordLength];
for(int k=0; k<wordTable.length; ++k)
{
c = word.charAt(k);
out.println(c);
}
}
out.close();
}
public static void main(String[] args) throws IOException
{
String nazwaPlikuWejsciowego = args[0];
String nazwaPlikuWyjsciowego = args[1];
Start doit = new Start(nazwaPlikuWejsciowego, nazwaPlikuWyjsciowego);
doit.saveing();
}
}
My problem is saving to file. After the saveing method above, the file does not contain any single character. When I move the out.close() to while for instance, the file contains one word. When out.close() is in for, the program saves one character only. Why?

add out.flush() before out.close().
You need to flush the bytes to disk before you close it..

This ((word = in.next()) != null) will throw an exception.
in.next() doesn't return null when there are no more elements. Take a look at the API

How do I concatenate sequential files in order with Java?

I have a directory that contains sequentially numbered log files and some Excel spreadsheets used for analysis. The log file are ALWAYS sequentially numbered beginning at zero, but the number of them can vary. I am trying to concatenate the log files, in the order they were created into a single text file which will be a concatenation of all the log files.
For instance, with log files foo0.log, foo1.log, foo2.log would be output to concatenatedfoo.log by appending foo1 after foo0, and foo2 after foo1.
I need to count all the files in the given directory with the extension of *.log, using the count to drive a for-loop that also generates the file name for concatenation. I'm having a hard time finding a way to count the files using a filter...none of the Java Turtorials on file operations seem to fit the situation, but I'm sure I'm missing something. Does this approach make sense? or is there an easier way?
int numDocs = [number of *.log docs in directory];
//
for (int i = 0; i <= numberOfFiles; i++) {
fileNumber = Integer.toString(i);
try
{
FileInputStream inputStream = new FileInputStream("\\\\Path\\to\\file\\foo" + fileNumber + ".log");
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream));
try
{
BufferedWriter metadataOutputData = new BufferedWriter(new FileWriter("\\\\Path\\to\\file\\fooconcat.log").append());
metadataOutputData.close();
}
//
catch (IOException e) // catch IO exception writing final output
{
System.err.println("Exception: ");
System.out.println("Exception: "+ e.getMessage().getClass().getName());
e.printStackTrace();
}
catch (Exception e) // catch IO exception reading input file
{
System.err.println("Exception: ");
System.out.println("Exception: "+ e.getMessage().getClass().getName());
e.printStackTrace();
}
}

how about
public static void main(String[] args){
final int BUFFERSIZE = 1024 << 8;
File baseDir = new File("C:\\path\\logs\\");
// Get the simple names of the files ("foo.log" not "/path/logs/foo.log")
String[] fileNames = baseDir.list(new FilenameFilter() {
#Override
public boolean accept(File dir, String name) {
return name.endsWith(".log");
}
});
// Sort the names
Arrays.sort(fileNames);
// Create the output file
File output = new File(baseDir.getAbsolutePath() + File.separatorChar + "MERGED.log");
try{
BufferedOutputStream out = new BufferedOutputStream(new FileOutputStream(output), BUFFERSIZE);
byte[] bytes = new byte[BUFFERSIZE];
int bytesRead;
final byte[] newLine = "\n".getBytes(); // use to separate contents
for(String s : fileNames){
// get the full path to read from
String fullName = baseDir.getAbsolutePath() + File.separatorChar + s;
BufferedInputStream in = new BufferedInputStream(new FileInputStream(fullName),BUFFERSIZE);
while((bytesRead = in.read(bytes,0,bytes.length)) != -1){
out.write(bytes, 0, bytesRead);
}
// close input file and ignore any issue with closing it
try{in.close();}catch(IOException e){}
out.write(newLine); // seperation
}
out.close();
}catch(Exception e){
throw new RuntimeException(e);
}
}
This code DOES assume that the "sequential naming" would be zero padded such that they will lexigraphically (?? sp) sort correctly. i.e. The files would be
0001.log (or blah0001.log, or 0001blah.log etc)
0002.log
....
0010.log
and not
1.log
2.log
...
10.log
The latter pattern will not sort correctly with the code I have given.

Here's some code for you.
File dir = new File("C:/My Documents/logs");
File outputFile = new File("C:/My Documents/concatenated.log");
Find the ".log" files:
File[] files = dir.listFiles(new FilenameFilter() {
#Override
public boolean accept(File file, String name) {
return name.endsWith(".log") && file.isFile();
}
});
Sort them into the appropriate order:
Arrays.sort(files, new Comparator<File>() {
#Override
public int compare(File file1, File file2) {
return numberOf(file1).compareTo(numberOf(file2));
}
private Integer numberOf(File file) {
return Integer.parseInt(file.getName().replaceAll("[^0-9]", ""));
}
});
Concatenate them:
byte[] buffer = new byte[8192];
OutputStream out = new BufferedOutputStream(new FileOutputStream(outputFile));
try {
for (File file : files) {
InputStream in = new FileInputStream(file);
try {
int charCount;
while ((charCount = in.read(buffer)) >= 0) {
out.write(buffer, 0, charCount);
}
} finally {
in.close();
}
}
} finally {
out.flush();
out.close();
}

By having the log folder as a File object, you can code like this
for (File logFile : logFolder.listFiles()){
if (logFile.getAbsolutePath().endsWith(".log")){
numDocs++;
}
}
to find the number of log files.

I would;
open the output file once. Just use a PrintWriter.
in a loop ...
create a File for each possible file
if it doesn't exist break the loop.
Using a BufferedReader
to read the lines of the file with readLine()
write each line to the output file.
You should be able to do this with about 12 lines of code. I would pass the IOExceptions to the caller.

You can use SequenceInputStream for concatenation of FileInputStreams.
To see all log files File.listFiles(FileFilter) can be used.
It will give you unsorted array with files. To sort files in right order, use Arrays.sort.
Code example:
static File[] logs(String dir) {
File root = new File(dir);
return root.listFiles(new FileFilter() {
#Override
public boolean accept(File pathname) {
return pathname.isFile() && pathname.getName().endsWith(".log");
}
});
}
static String cat(final File[] files) throws IOException {
Enumeration<InputStream> e = new Enumeration<InputStream>() {
int index;
#Override
public boolean hasMoreElements() {
return index < files.length;
}
#Override
public InputStream nextElement() {
index++;
try {
return new FileInputStream(files[index - 1]);
} catch (FileNotFoundException ex) {
throw new RuntimeException("File not available!", ex);
}
}
};
SequenceInputStream input = new SequenceInputStream(e);
StringBuilder sb = new StringBuilder();
int c;
while ((c = input.read()) != -1) {
sb.append((char) c);
}
return sb.toString();
}
public static void main(String[] args) throws IOException {
String dir = "<path-to-dir-with-logs>";
File[] logs = logs(dir);
for (File f : logs) {
System.out.println(f.getAbsolutePath());
}
System.out.println();
System.out.println(cat(logs));
}

Trouble with java snippets

public class GenericWorldLoader implements WorldLoader {
#Override
public LoginResult checkLogin(PlayerDetails pd) {
Player player = null;
int code = 2;
File f = new File("data/savedGames/" + NameUtils.formatNameForProtocol(pd.getName()) + ".dat.gz");
if(f.exists()) {
try {
InputStream is = new GZIPInputStream(new FileInputStream(f));
String name = Streams.readRS2String(is);
String pass = Streams.readRS2String(is);
if(!name.equals(NameUtils.formatName(pd.getName()))) {
code = 3;
}
if(!pass.equals(pd.getPassword())) {
code = 3;
}
} catch(IOException ex) {
code = 11;
}
}
if(code == 2) {
player = new Player(pd);
}
return new LoginResult(code, player);
}
#Override
public boolean savePlayer(Player player) {
try {
OutputStream os = new GZIPOutputStream(new FileOutputStream("data/savedGames/" + NameUtils.formatNameForProtocol(player.getName()) + ".dat.gz"));
IoBuffer buf = IoBuffer.allocate(1024);
buf.setAutoExpand(true);
player.serialize(buf);
buf.flip();
byte[] data = new byte[buf.limit()];
buf.get(data);
os.write(data);
os.flush();
os.close();
return true;
} catch(IOException ex) {
return false;
}
}
#Override
public boolean loadPlayer(Player player) {
try {
File f = new File("data/savedGames/" + NameUtils.formatNameForProtocol(player.getName()) + ".dat.gz");
InputStream is = new GZIPInputStream(new FileInputStream(f));
IoBuffer buf = IoBuffer.allocate(1024);
buf.setAutoExpand(true);
while(true) {
byte[] temp = new byte[1024];
int read = is.read(temp, 0, temp.length);
if(read == -1) {
break;
} else {
buf.put(temp, 0, read);
}
}
buf.flip();
player.deserialize(buf);
return true;
} catch(IOException ex) {
return false;
}
}
}
Yeah so... My problem is that this seems to save 'something' in really complex and hard to read way(binary) and I'd rather have it as an .txt, in easily readable format. how to convert?
EDIT: I'm not using Apache Mina, so what should I replace
IoBuffer buf = IoBuffer.allocate(1024);
buf.setAutoExpand(true);"
with?

checkLogin() obviously checks whether the specified login has matching data present and whether the password is correct.
savePlayer() method saves the player.
loadPlayer() loads it again.
The data format used is gzip (wiki) and it is written as a stream of serialized data. If you want to make it more readable, you might want to overload (or just use it, if it is good) toString() method of Player class and to write player.toString() into a new text file using e.g. BufferedWriter wrapped around a File Writer:
String playerName = NameUtils.formatNameForProtocol(player.getName());
BufferedWriter writer = new BufferedWriter(new FileWriter(playerName + ".txt"));
writer.write(player.toString());
writer.close();