I need to loop through a .txt document called archive.txt until all data is in the document is exhausted.
I'm trying to return the data in a multidimensional array that creates a new row for every eighth data point.
So far, I have managed to loop through the data but can not seem to organise it.
Below is the function that can only spit out the data line per line.
private void findContract() {
Scanner input = null; // this is to keep the compiler happy
// as the object initialisation is in a separate block
try {
input = new Scanner(new File("archive.txt"));
} catch (FileNotFoundException e) {
System.out.println("File doesn't exist");
System.exit(1);
}
while (input.hasNext()) {
String dDate = input.next();
System.out.println(dDate);
}
input.close();
}
Example of first 8 data points from text file (archive.txt)
15-Sep-2015 2 1 12 N MT230N 617 CMcgee
The outcome of all of this is I need to be able to select a data point by row & column.
If anyone could show me the correct way it would be hugely appreciated. I have tried several methods & the above function is the last instance where it displays the data from the file.
With java 8 you can use this for any delimiter you may be using.
public static String[][] fileToMatriz(String file, String delimiter) throws IOException {
try (Stream<String> stream = Files.lines(Paths.get(file))) {
return stream.map(s -> s.split(delimiter))
.toArray(String[][]::new);
}
}
Related
I have this code which is used to read lines from a file and insert it into Postgre:
try {
BufferedReader reader;
try {
reader = new BufferedReader(new FileReader(
"C:\\in_progress\\test.txt"));
String line = reader.readLine();
while (line != null) {
System.out.println(line);
Thread.sleep(100);
Optional<ProcessedWords> isFound = processedWordsService.findByKeyword(line);
if(!isFound.isPresent()){
ProcessedWords obj = ProcessedWords.builder()
.keyword(line)
.createdAt(LocalDateTime.now())
.build();
processedWordsService.save(obj);
}
// read next line
line = reader.readLine();
}
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
catch (Exception e) {
e.printStackTrace();
}
How I can remove a line from the file after the line is inserted into SQL database?
The issues with the current code:
Adhere to the Single responsibility principle. Your code is doing too many things: reads from a file, performs findByKeyword() call, prepares the data and hands it out to store in the database. It's hardly can be thoroughly tested, and it's very difficult to maintain.
Always use try-with-recourses to get your recourses closed at any circumstances.
Don't catch the general Exception type - your code should only catch thous exceptions, which are more or less expected and for which there's a clear scenario on how to handle them. But don't catch all the exceptions.
How I can remove a line from the file after the line is inserted into SQL database?
It is not possible to remove a line from a file in the literal sense. You can override the contents of the file or replace it with another file.
My advice would be to file data in memory, process it, and then write the lines which should be retained into the same file (i.e. override the file contents).
You can argue that the file is huge and dumping it into memory would result in an OutOfMemoryError. And you want to read a line from a file, process it somehow, then store the processed data into the database and then write the line into a file... So that everything is done line by line, all actions in one go for a single line, and as a consequence all the code is crammed in one method. I hope that's not the case because otherwise it's a clear XY-problem.
Firstly, File System isn't a reliable mean of storing data, and it's not very fast. If the file is massive, then reading and writing it will a take a considerable amount of time, and it's done just it in order to use a tinny bit of information then this approach is wrong - this information should be stored and structured differently (i.e. consider placing into a DB) so that it would be possible to retrieve the required data, and there would be no problem with removing entries that are no longer needed.
But if the file is lean, and it doesn't contain critical data. Then it's totally fine, I will proceed assuming that it's the case.
The overall approach is to generate a map Map<String, Optional<ProcessedWords>> based on the file contents, process the non-empty optionals and prepare a list of lines to override the previous file content.
The code below is based on the NIO2 file system API.
public void readProcessAndRemove(ProcessedWordsService service, Path path) {
Map<String, Optional<ProcessedWords>> result;
try (var lines = Files.lines(path)) {
result = processLines(service, lines);
} catch (IOException e) {
result = Collections.emptyMap();
logger.log();
e.printStackTrace();
}
List<String> linesToRetain = prepareAndSave(service, result);
writeToFile(linesToRetain, path);
}
Processing the stream of lines from a file returned Files.lines():
private static Map<String, Optional<ProcessedWords>> processLines(ProcessedWordsService service,
Stream<String> lines) {
return lines.collect(Collectors.toMap(
Function.identity(),
service::findByKeyword
));
}
Saving the words for which findByKeyword() returned an empty optional:
private static List<String> prepareAndSave(ProcessedWordsService service,
Map<String, Optional<ProcessedWords>> wordByLine) {
wordByLine.forEach((k, v) -> {
if (v.isEmpty()) saveWord(service, k);
});
return getLinesToRetain(wordByLine);
}
private static void saveWord(ProcessedWordsService service, String line) {
ProcessedWords obj = ProcessedWords.builder()
.keyword(line)
.createdAt(LocalDateTime.now())
.build();
service.save(obj);
}
Generating a list of lines to retain:
private static List<String> getLinesToRetain(Map<String, Optional<ProcessedWords>> wordByLine) {
return wordByLine.entrySet().stream()
.filter(entry -> entry.getValue().isPresent())
.map(Map.Entry::getKey)
.collect(Collectors.toList());
}
Overriding the file contents using Files.write(). Note: since varargs OpenOption isn't provided with any arguments, this call would be treated as if the CREATE, TRUNCATE_EXISTING, and WRITE options are present.
private static void writeToFile(List<String> lines, Path path) {
try {
Files.write(path, lines);
} catch (IOException e) {
logger.log();
e.printStackTrace();
}
}
For Reference
import java.io.*;
public class RemoveLinesFromAfterProcessed {
public static void main(String[] args) throws Exception {
String fileName = "TestFile.txt";
String tempFileName = "tempFile";
File mainFile = new File(fileName);
File tempFile = new File(tempFileName);
try (BufferedReader br = new BufferedReader(new FileReader(mainFile));
PrintWriter pw = new PrintWriter(new FileWriter(tempFile))
) {
String line;
while ((line = br.readLine()) != null) {
if (toProcess(line)) { // #1
// process the code and add it to DB
// ignore the line (i.e, not add to temp file)
} else {
// add to temp file.
pw.write(line + "\n"); // #2
}
}
} catch (Exception e) {
e.printStackTrace();
}
// delete the old file
boolean hasDeleted = mainFile.delete(); // #3
if (!hasDeleted) {
throw new Exception("Can't delete file!");
}
boolean hasRenamed = tempFile.renameTo(mainFile); // #4
if (!hasRenamed) {
throw new Exception("Can't rename file!");
}
System.out.println("Done!");
}
private static boolean toProcess(String line) {
// any condition
// sample condition for example
return line.contains("aa");
}
}
Read the file.
1: The condition to decide whether to delete the line or to retain it.
2: Write those line which you don't want to delete into the temporary file.
3: Delete the original file.
4: Rename the temporary file to original file name.
The basic idea is the same as what #Shiva Rahul said in his answer.
However another approach can be , store all the line numbers you want to delete in a list. After you have all the required line numbers that you want to delete you can use LineNumberReader to check and duplicate your main file.
Mostly I have used this technique in batch-insert where I was unsure how many lines may have a particular file plus before removal of lines had to do lot of processing.
It may not be suitable for your case ,just posting the suggestion here if any one bumps to this thread.
private void deleteLines(String inputFilePath,String outputDirectory,List<Integer> lineNumbers) throws IOException{
File tempFile = new File("temp.txt");
File inputFile = new File(inputFilePath);
// using LineNumberReader we can fetch the line numbers of each line
LineNumberReader lineReader = new LineNumberReader(new FileReader(inputFile));
//writter for writing the lines into new file
BufferedWriter bufferedWriter = new BufferedWriter(new FileWriter(tempFile));
String currentLine;
while((currentLine = lineReader.readLine()) != null){
//if current line number is present in removeList then put empty line in new file
if(lineNumbers.contains(lineReader.getLineNumber())){
currentLine="";
}
bufferedWriter.write(currentLine + System.getProperty("line.separator"));
}
//closing statements
bufferedWriter.close();
lineReader.close();
//delete the main file and rename the tempfile to original file Name
boolean delete = inputFile.delete();
//boolean b = tempFile.renameTo(inputFile); // use this to save the temp file in same directory;
boolean b = tempFile.renameTo(new File(outputDirectory+inputFile.getName()));
}
To use this function all you have to do is gather all the required line numbers.inputFilePath is the path of the source file and outputDirectory is where I want store the file after processing.
I have this app where i process a very large file, Extract the lines that have the same first 5 characters (i call this currentlineId ), use them to create an object and do something with it, example sample of the file contents:
AZDFS12345678998765432345678
AZDFS09876545432345678987654
AZDFS34568987654567890987654
AZDFS12345670987654345678998
AZDFS12345098734567765123456
// the lines above have the same first 5 characters, they create Object1.
FGHJUY121324
FGHJUY089909
FGHJUYTTUTUU
//same for the lines above, they create Object2.
NB: the lines will always be in an order where lines with the same first 5 will always be together (abover/below each other) so i wonn't have lines all over the place
My current function code:
private void processScpFile(File file) {
LOGGER.info("Processing File: {} ", file.getName());
try (var br = new BufferedReader(new FileReader(file))) {
String currentLine;
String lastLineId = null;
List<String> similarLineIdsList = new ArrayList<>();
while ((currentLine = br.readLine()) != null) {
if (StringUtils.isEmpty(lastLineId)) {
lastLineId = currentLine.substring(0,5);
}
if (lastLineId.equals(currentLine.substring(0,5))) {
similarLineIdsList.add(currentLine);
}
else if (!lastLineId.equals(currentLine.substring(0,5))) {
doSomethinsWithTheList(similarLineIdsList);
similarLineIdsList.clear();
similarLineIdsList.add(currentLine);
lastLineId= currentLine.substring(0,5);
}
}
doSomethinsWithTheList(similarLineIdsList);
}
catch (IOException e) {
LOGGER.error("Couldn't read file, {}", e.getMessage(), e);
}
}
Now this has worked well up until now, but going forward i have to process files where i would have for instance over 100k lines with same first 5, which makes this process very slow.
Please do you have any suggestion on haow to make this process faster, thank you
Edit: just to be precise it's the generating the list with the same first 5 chars that's slower as the number of similar lines gets larger.
I am using a .txt file that has data in it formatted the same way throughout.
For example:
title|format|onLoan|loanedTo|loanedOn
title|format|onLoan|loanedTo|loanedOn
title|format|onLoan|loanedTo|loanedOn
title|format|onLoan|loanedTo|loanedOn
I am opening the file and then trying to transfer that information into class object and then put that object in a arrayList of that class.
The problem I am having is with the age old ArrayIndexOutOfBoundsException. I am not sure what is causing it. It is probably something simple. Any guidance would be appreciated.
java.lang.ArrayIndexOutOfBoundsException: 1
at Library.open(Library.java:230)
Scanner input = null;
String mediaItemString = "";
MediaItem open = new MediaItem(); //class created for this object
String[] libraryItem;
//try catch block for exception handling
try {
input = new Scanner(new File("library.txt"));
while (input.hasNextLine()) {
mediaItemString = input.nextLine();
libraryItem = mediaItemString.split("\\|");
System.out.println(libraryItem[0].toString());
System.out.println(libraryItem[1].toString()); //error here, line 230
System.out.println(Boolean.parseBoolean(libraryItem[2].toString()));
System.out.println(libraryItem[3].toString());
System.out.println(libraryItem[4].toString());
open.setTitle(libraryItem[0].toString());
open.setFormat(libraryItem[1].toString());
open.setOnLoan(Boolean.parseBoolean(libraryItem[2].toString()));
open.setLoanedTo(libraryItem[3].toString());
open.setDateLoaned(libraryItem[4].toString());
items.add(open);
}
} catch (FileNotFoundException e){
System.out.println("There was an error with the file.");
} finally {
input.close();
}
Well I was hoping to split the string with delimiters and then assign those values to the MediaItem appropriately.
Make sure you don't have any missing value for a column. Also check for empty new lines.
It looks like some lines does not follow described format.
My suggection - add condition before array processing like
...
libraryItem = mediaItemString.split("\\|");
if(libraryItem.length <5) {
log.error("Error: libraryItem.length is {} for string {}", libraryItem.length, mediaItemString);
continue;
}
System.out.println(libraryItem[0].toString());
...
I'm working on a program that needs to update a line that depends its value on the result of a line that goes read after. I thought that I could use two BufferedReaders in Java to position the reader on the line to update while the other one goes for the line that fixes the value (it can be an unknown number of lines ahead). The problem here is that I'm using two BufferedReaders on the same file and even if I think I'm doing right with the indexes the result in debug doesn't seem to be reliable.
Here's the code:
String outFinal
FileName=fileOut;
File fileDest=new File(outFinalFileName);
try {
fout = new BufferedWriter(
new OutputStreamWriter(
new FileOutputStream(fileDest)));
} catch (FileNotFoundException e) {
e.printStackTrace();
}
FileReader inputFile=null;
try {
inputFile = new FileReader(inFileName);
} catch (FileNotFoundException e2) {
e2.printStackTrace();
}
BufferedReader fin = new BufferedReader(inputFile);
BufferedReader finChecker = new BufferedReader(inputFile); //Checks the file and matches record to change
String line="";
String lineC="";
int lineNumber=0;
String recordType="";
String statusCode="";
try {
while ((lineC = finChecker.readLine()) != null) {
lineNumber++;
if (lineNumber==1)
line=fin.readLine();
recordType=lineC.substring(0,3);//Gets current Record Type
if (recordType.equals("35")){
while(!line.equals(lineC)){
line=fin.readLine();
if (line==null)
break;
fout.write(line);
}
}else if (recordType.equals("32")){
statusCode=lineC.substring(4,7);
if(statusCode.equals("XX")){
updateRecordLine(line,fout);
}
}
}
returnVal=true;
} catch (IOException e) {
e.printStackTrace();
}
Thanks in advance.
Well, the BufferedReader only reads stuff, it doesn't have the ability to write data back out. So, what you would need is a BufferedReader to get stuff in, and a BufferedWriter that takes all the input from the BufferedReader, and outputs it to a temp file, with the corrected/appended data.
Then, when you're done (i.e. both BufferedReader and BufferedWriter streams are closed), you need to either discard the original file, or rename the temp file to the name of the original file.
You are basically copying the original file to a temp file, modifying the line in question in the temp file's output, and then copying/renaming the temp file over the original.
ok, i see some problem in your code exactly on these lines-->
recordType=lineC.substring(0,3);//Gets current Record Type
if (recordType.equals("35")){
if you see on the first line, you are getting the substring of recordType into recordType. Now recordType length is 3. If at all the recordType has only 2 characters, then substring throws arrayIndexOutOfBoundsException. So when no runtime exceptions, its length is 3 and on the next line you are calling the equals method that has a string with 2 characters.
Will this if block ever run ?
I am writing a program in Java that requires me to compare the data in 2 files. I have to check each line from file 1 against each line of file 2 and if I find a match write them to a third file. After I read to the end of file 2, how do I reset the pointer to the beginning of the file?
public class FiFo {
public static void main(String[] args)
{
FileReader file1=new FileReader("d:\\testfiles\\FILE1.txt");
FileReader file2=new FileReader("d:\\testfiles\\FILE2.txt");
try{
String s1,s2;
while((s1=file1.data.readLine())!=null){
System.out.println("s1: "+s1);
while((s2=file2.data.readLine())!=null){
System.out.println("s2: "+s2);
}
}
file1.closeFile();
file2.closeFile();
}catch (IOException e) {
e.printStackTrace();
}
}
}
class FileReader {
BufferedReader data;
DataInputStream in;
public FileReader(String fileName)
{
try{
FileInputStream fstream = new FileInputStream(fileName);
data = new BufferedReader(new InputStreamReader(fstream));
}
catch (IOException e) {
e.printStackTrace();
}
}
public void closeFile()
{
try{
in.close();
}
catch (IOException e) {
e.printStackTrace();
}
}
}
I believe RandomAccessFile is what you need. It contains: RandomAccessFile#seek and RandomAccessFile#getFilePointer.
rewind() is seek(0)
I think the best thing to do would be to put each line from file 1 into a HashMap; then you could check each line of file 2 for membership in your HashMap rather than reading through the entire file once for each line of file 1.
But to answer your question of how to go back to the beginning of the file, the easiest thing to do is to open another InputStream/Reader.
Obviously you could just close and reopen the file like this:
while((s1=file1.data.readLine())!=null){
System.out.println("s1: "+s1);
FileReader file2=new FileReader("d:\\testfiles\\FILE2.txt");
while((s2=file2.data.readLine())!=null){
System.out.println("s2: "+s2);
//compare s1 and s2;
}
file2.closeFile()
}
But you really don't want to do it that way, since this algorithm's running time is O(n2). if there were 1000 lines in file A, and 10000 lines in file B, your inner loop would run 1,000,000 times.
What you should do is read each line and store it in a collection that allows quick checks to see if an item is already contained(probably a HashSet).
If you only need to check to see that every line in file 2 is in file 1, then you just add each line in file one to a HashSet, and then check to see that every line in file 2 is in that set.
If you need to do a cross comparison where you find every string that's in one but not the other, then you'll need two hash sets, one for each file. (Although there's a trick you could do to use just one)
If the files are so large that you don't have enough memory, then your original n2 method would never have worked anyway.
well, Gennady S. answer is what I would use to solve your problem.
I am writing a program in Java that requires me to compare the data in 2 files
however, I would rather not code this up again.. I would rather use something like http://code.google.com/p/java-diff-utils/
As others have suggested, you should consider other approaches to the problem. For the specific question of returning to a previous point in a file, java.io.FileReader would appear to inherit mark() and reset() methods that address this goal. Unfortunately, markSupported() returns false.
Alternatively, BufferedReader does support mark(). The program below prints true, illustrating the effect.
package cli;
import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
public class FileReaderTest {
public static void main(String[] args) {
try {
BufferedReader in = new BufferedReader(new InputStreamReader(
new FileInputStream("src/cli/FileReaderTest.java")));
in.mark(1);
int i1 = in.read(); in.read(); in.read();
in.reset();
int i2 = in.read();
System.out.println(i1 == i2);
} catch (IOException e) {
e.printStackTrace(System.err);
}
}
}
As noted, there are better algorithms - investigate these
aside:
FileReader doesn't implement mark and reset, so trashgod's comments are inaccurate.
You'd either have to implement a version of this (using RandomAccessFile or what not) or wrap in a BufferedReader. However, the latter will load the whole thing in memory if you mark it
Just a quick Question. can't you keep one object pointed at the start of the file and traverse through the file with another object? Then when you get to the end just point it to the object at the beginning of the file(stream). I believe C++ has such mechanisms with file I/O ( or is it stream I/O)
I believe that you could just re-initialize the file 2 file reader and that should reset it.
If you can clearly indentify the dimension of your file you can use mark(int readAheadLimit) and reset() from the class BufferedReader.
The method mark(int readAhedLimit) add a marker to the current position of your BufferedReader and you can go back to the marker using reset().
Using them you have to be careful to the number of characters to read until the reset(), you have to specify them as the argument of the function mark(int readAhedLimit).
Assuming a limit of 100 characters your code should look like:
class MyFileReader {
BufferedReader data;
int maxNumberOfCharacters = 100;
public MyFileReader(String fileName)
{
try{
FileInputStream fstream = new FileInputStream(fileName);
data = new BufferedReader(new InputStreamReader(fstream));
//mark the current position, in this case the beginning of the file
data.mark(maxNumberOfCharacters);
}
catch (IOException e) {
e.printStackTrace();
}
}
public void resetFile(){
data.reset();
}
public void closeFile()
{
try{
in.close();
}
catch (IOException e) {
e.printStackTrace();
}
}
}
If you just want to reset the file pointer to the top of the file, reinitialize your buffer reader. I assume that you are also using the try and catch block to check for end of the file.
`//To read from a file.
BufferedReader read_data_file = new BufferedReader(new FileReader("Datafile.dat"));'
Let's say this is how you have your buffer reader defined. Now, this is how you can check for end of file=null.
boolean has_data= true;
while(has_data)
{
try
{
record = read_data_file.readLine();
delimit = new StringTokenizer(record, ",");
//Reading the input in STRING format.
cus_ID = delimit.nextToken();
cus_name = delimit.nextToken();'
//And keep grabbing the data and save it in appropriate fields.
}
catch (NullPointerException e)
{
System.out.println("\nEnd of Data File... Total "+ num_of_records
+ " records were printed. \n \n");
has_data = false; //To exit the loop.
/*
------> This point is the trouble maker. Your file pointer is pointing at the end of the line.
-->If you want to again read all the data FROM THE TOP WITHOUT RECOMPILING:
Do this--> Reset the buffer reader to the top of the file.
*/
read_data_file = new BufferedReader(new FileReader(new File("datafile.dat")));
}
By reinitializing the buffer reader you will reset the file reader mark/pointer to the top of the file and you won't have to recompile the file to set the file reader marker/pointer to beginning/top of the file.
You need to reinitialize the buffer reader only if you don't want to recompile and pull off the same stunt in the same run. But if you wish to just run loop one time then you don't have to all this, by simply recompiling the file, the file reader marker will be set to the top/beginning of the file.