We Keep Coding

Merging sorted Files using multithreading

Multithreading is new to me so sorry for mistakes.
I have written the below program which merges files with mulithreading but I am not able to figure out how to manage the last file and after one iteration how to merge the newly created files.
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.FileWriter;
import java.util.ArrayList;
public class MergerSorter extends Thread {
int fileNumber = 1;
public static void main(String[] args) {
startMergingfiles(9);
}
public MergerSorter(int fileNum) {
fileNumber = fileNum;
}
public static void startMergingfiles(int numberOfFiles) {
int objectcounter = 0;
while (numberOfFiles != 1) {
try {
ArrayList<MergerSorter> objectList = new ArrayList<MergerSorter>();
for (int j = 1; j <= numberOfFiles; j = j + 2) {
if (numberOfFiles == j) {// Last Single remaining File
} else {
objectList.add(new MergerSorter(j));
objectList.get(objectcounter).start();
objectList.get(objectcounter).join();
objectcounter++;
}
}
objectcounter = 0;
numberOfFiles = numberOfFiles / 2;
} catch (Exception e) {
System.out.println(e);
}
}
}
public void run() {
try {
FileReader fileReader1 = new FileReader("src/externalsort/" + Integer.toString(fileNumber));
FileReader fileReader2 = new FileReader("src/externalsort/" + Integer.toString(fileNumber + 1));
BufferedReader bufferedReader1 = new BufferedReader(fileReader1);
BufferedReader bufferedReader2 = new BufferedReader(fileReader2);
String line1 = bufferedReader1.readLine();
String line2 = bufferedReader2.readLine();
FileWriter tmpFile = new FileWriter("src/externalsort/" + Integer.toString(fileNumber) + "op.txt", false);
int whichFileToRead = 0;
boolean file_1_reader = true;
boolean file_2_reader = true;
while (file_1_reader || file_2_reader) {
if (file_1_reader == false) {
tmpFile.write(line2 + "\r\n");
whichFileToRead = 2;
} else if (file_2_reader == false) {
tmpFile.write(line1 + "\r\n");
whichFileToRead = 1;
} else {
String value1 = line1.substring(0, 10);
String value2 = line2.substring(0, 10);
int ans = value1.compareTo(value2);
if (ans < 0) {
tmpFile.write(line1 + "\r\n");
whichFileToRead = 1;
} else if (ans > 0) {
tmpFile.write(line2 + "\r\n");
whichFileToRead = 2;
} else if (ans == 0) {
tmpFile.write(line1 + "\r\n");
whichFileToRead = 1;
}
}
if (whichFileToRead == 1) {
line1 = bufferedReader1.readLine();
if (line1 == null)
file_1_reader = false;
} else {
line2 = bufferedReader2.readLine();
if (line2 == null)
file_2_reader = false;
}
}
tmpFile.close();
bufferedReader1.close();
bufferedReader2.close();
fileReader1.close();
fileReader2.close();
} catch (Exception e) {
System.out.println(e);
}
}
}
I am trying to merge sorted files with multithreading. Say I have 50 files and I want to merge all these individual files into one final sorted file but I want to speed up and utilize every core by multi threading but I am not able to do it. And the files are big so they can't be placed in heap/RAM so I have to read every file and keep writing.

You can do this with merge sort, but instead of lots of little sorted lists, you'll need to use lots of little sorted files. Once you have broken all of the files down into small sorted files, you can start merging them together again until you end up with a single sorted file.
Unfortunately, you likely won't be able to achieve high CPU utilisation as much of the time will be spend waiting for disk I/O to complete.
Edit: just read your response to a comment and it sounds like you are asking for help on the last step of the merge sort. The graphics in the wiki link above will also help you understand. So, assuming all of your files are sorted, here we go:
Read 1 item from each file
Figure out which lowest/smallest/whatever and write that line to the result file
Read a new item from the file which just provided the last item
Repeat steps 2 and 3 until all files have been completely read.

JAVA read text files, count numbers and write it to Jtable [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I am still learning JAVA and have been trying to find a solution for my program for a few days, but I haven't gotten it fixed yet.
I have many text files (my program saves). The files look like this:
text (tab) number (tab) number (tab)...
text (tab) number (tab) number (tab)...
(tab) means that there is tabulation mark,
text means that is text (string),
number means that there is number (integer).
number of files can be from 1 up to 32 and file with names like: january1; january2; january3...
I need to read all of those files (ignore strings) and sum only numbers like so:
while ((line = br.readLine()) != null) {
counter=counter+1;
String[] info = line.split("\\s+");
for(int j = 2; j < 8; j++) {
int num = Integer.parseInt(info[j]);
data[j][counter]=data[j][counter]+num;
}
};
Simply I want sum all that "tables" to array of arrays (or to any similar kind of variable) and then display it as table. If someone knows any solution or can link any similar calculation, that would be awesome!

So, as I see it, you have four questions you need answered, this goes against the site etiquette of asking A question, but will give it a shot
How to list a series of files, presumably using some kind of filter
How to read a file and process the data in some meaningful way
How to manage the data in data structure
Show the data in a JTable.
Listing files
Probably the simplest way to list files is to use File#list and pass a FileFilter which meets your needs
File[] files = new File(".").listFiles(new FileFilter() {
#Override
public boolean accept(File pathname) {
return pathname.getName().toLowerCase().startsWith("janurary");
}
});
Now, I'd write a method which took a File object representing the directory you want to list and a FileFilter to use to search it...
public File[] listFiles(File dir, FileFilter filter) throws IOException {
if (dir.exists()) {
if (dir.isDirectory()) {
return dir.listFiles(filter);
} else {
throw new IOException(dir + " is not a valid directory");
}
} else {
throw new IOException(dir + " does not exist");
}
}
This way you could search for a number of different set of files based on different FileFilters.
Of course, you could also use the newer Paths/Files API to find files as well
Reading files...
Reading multiple files comes down to the same thing, reading a single file...
// BufferedReader has a nice readline method which makes
// it easier to read text with. You could use a Scanner
// but I prefer BufferedReader, but that's me...
try (BufferedReader br = new BufferedReader(new FileReader(new File("...")))) {
String line = null;
// Read each line
while ((line = br.readLine()) != null) {
// Split the line into individual parts, on the <tab> character
String parts[] = line.split("\t");
int sum = 0;
// Staring from the first number, sum the line...
for (int index = 1; index < parts.length; index++) {
sum += Integer.parseInt(parts[index].trim());
}
// Store the key/value pairs together some how
}
}
Now, we need some way to store the results of the calculations...
Have a look at Basic I/O for more details
Managing the data
Now, there are any number of ways you could do this, but since the amount of data is variable, you want a data structure that can grow dynamically.
My first thought would be to use a Map, but this assumes you want to combining rows with the same name, otherwise you should just us a List within a List, where the outer List represents the rows and the Inner list represents the column values...
Map<String, Integer> data = new HashMap<>(25);
File[] files = listFiles(someDir, januraryFilter);
for (File file : files {
readFile(file, data);
}
Where readFile is basically the code from before
protected void readData(File file, Map<String, Integer> data) throws IOException {
try (BufferedReader br = new BufferedReader(new FileReader(file))) {
String line = null;
// Read each line
while ((line = br.readLine()) != null) {
//...
// Store the key/value pairs together some how
String name = parts[0];
if (data.containsKey(name)) {
int previous = data.get(name);
sum += previous;
}
data.put(name, sum);
}
}
}
Have a look at the Collections Trail for more details
Showing the data
And finally, we need to show the data. You could simply use a DefaultTableModel, but you already have the data in structure, why not re-use it with a custom TableModel
public class SummaryTableModel extends AbstractTableModel {
private Map<String, Integer> data;
private List<String> keyMap;
public SummaryTableModel(Map<String, Integer> data) {
this.data = new HashMap<>(data);
keyMap = new ArrayList<>(data.keySet());
}
#Override
public int getRowCount() {
return data.size();
}
#Override
public int getColumnCount() {
return 2;
}
#Override
public Class<?> getColumnClass(int columnIndex) {
Class type = Object.class;
switch (columnIndex) {
case 0:
type = String.class;
break;
case 1:
type = Integer.class;
break;
}
return type;
}
#Override
public Object getValueAt(int rowIndex, int columnIndex) {
Object value = null;
switch (columnIndex) {
case 0:
value = keyMap.get(rowIndex);
break;
case 1:
String key = keyMap.get(rowIndex);
value = data.get(key);
break;
}
return value;
}
}
Then you would simply apply it to a JTable...
add(new JScrollPane(new JTable(new SummaryTableModel(data)));
Take a look at How to Use Tables for more details
Conclusion
There are a lot of assumptions that have to be made which are missing from the context of the question; does the order of the files matter? Do you care about duplicate entries?
So it becomes near impossible to provide a single "answer" which will solve all of your problems

I took all the january1 january2... files from the location and used your same function to calculate the value to be stored.
Then I created a table with two headers, Day and Number. Then just added rows according to the values generated.
DefaultTableModel model = new DefaultTableModel();
JTable table = new JTable(model);
String line;
model.addColumn("Day");
model.addColumn("Number");
BufferedReader br = null;
model.addRow(new Object[]{"a","b"});
for(int i = 1; i < 32; i++)
{
try {
String sCurrentLine;
String filename = "january"+i;
br = new BufferedReader(new FileReader("C:\\january"+i+".txt"));
int counter = 0;
while ((sCurrentLine = br.readLine()) != null) {
counter=counter+1;
String[] info = sCurrentLine.split("\\s+");
int sum = 0;
for(int j = 2; j < 8; j++) {
int num = Integer.parseInt(info[j]);
sum += num;
}
model.addRow(new Object[]{filename, sum+""});
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (br != null)br.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
JFrame f = new JFrame();
f.setSize(300, 300);
f.add(new JScrollPane(table));
f.setVisible(true);

Use Labled Loop and Try-Catch. Below piece adds all number in a line.
You could get some hint from here:
String line = "text 1 2 3 4 del";
String splitLine[] = line.split("\t");
int sumLine = 0;
int i = 0;
contSum: for (; i < splitLine.length; i++) {
try {
sumLine += Integer.parseInt(splitLine[i]);
} catch (Exception e) {
continue contSum;
}
}
System.out.println(sumLine);

Here is another example using vectors . in this example directories will be searched for ".txt" files and added to the JTable.
The doIt method will take in the folder where your text files are located.
this will then with recursion, look for files in folders.
each file found will be split and added following you example file.
public class FileFolderReader
{
private Vector<Vector> rows = new Vector<Vector>();
public static void main(String[] args)
{
FileFolderReader fileFolderReader = new FileFolderReader();
fileFolderReader.doIt("D:\\folderoffiles");
}
private void doIt(String path)
{
System.out.println(findFile(new File(path)) + " in total");
JFrame frame = new JFrame();
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
Vector<String> columnNames = new Vector<String>();
columnNames.addElement("File Name");
columnNames.addElement("Size");
JTable table = new JTable(rows, columnNames);
JScrollPane scrollPane = new JScrollPane(table);
frame.add(scrollPane, BorderLayout.CENTER);
frame.setSize(300, 150);
frame.setVisible(true);
}
private int findFile(File file)
{
int totalPerFile = 0;
int total = 0;
File[] list = file.listFiles(new FilenameFilter()
{
public boolean accept(File dir, String fileName)
{
return fileName.endsWith(".txt");
}
});
if (list != null)
for (File textFile : list)
{
if (textFile.isDirectory())
{
total = findFile(textFile);
}
else
{
totalPerFile = scanFile(textFile);
System.out.println(totalPerFile + " in " + textFile.getName());
Vector<String> rowItem = new Vector<String>();
rowItem.addElement(textFile.getName());
rowItem.addElement(Integer.toString(totalPerFile));
rows.addElement(rowItem);
total = total + totalPerFile;
}
}
return total;
}
public int scanFile(File file)
{
int sum = 0;
Scanner scanner = null;
try
{
scanner = new Scanner(file);
while (scanner.hasNextLine())
{
String line = scanner.nextLine();
String[] info = line.split("\\s+");
int count = 1;
for (String stingInt : info)
{
if (count != 1)
{
sum = sum + Integer.parseInt(stingInt);
}
count++;
}
}
scanner.close();
}
catch (FileNotFoundException e)
{
// you will need to handle this
// don't do this !
e.printStackTrace();
}
return sum;
}
}

Java: Help outputting to a file?

Working on a Java project for a class. Professor wants us to output to a file, which I've never done before.
Instructions:
Create program1.states.in.out and include it in your project folder when you submit your project.
This file is NOT to appear like the displayed output in Step 3. It is to consist of only the
unformatted data lines (not spaced out; no headers….). The output file, then, will consist of fifty
data lines only. You will need this created copy for your input again ahead in Step 6. So
ensure it is formatted correctly. It will look like:
Texas Austin TX19759614Southwest 5
New_Mexico Santa_Fe NM 1736931Southwest 5
Arizona Phoenix AZ 4668631Southwest 5
This looks to be the same structure from his input file. But I've already trimmed/parsed that to fill the State array. How would I get it back into this format, then output it to a file?
Code used to trim/parse from the input file (in main):
inputString = br1.readLine();
while (inputString != null) {
stateName = inputString.substring(0, 15).trim();
stateCapital = inputString.substring(15, 30).trim();
stateAbbrev = inputString.substring(30, 32).trim();
statePop = Integer.parseInt(inputString.substring(32, 40).trim());
stateRegion = inputString.substring(40, 55).trim();
stateRegionNum = Integer.parseInt(inputString.substring(55));
Sdriver.insert(stateName, stateCapital, stateAbbrev, statePop, stateRegion, stateRegionNum);
inputString = br1.readLine(); // read next input line.
}
br1.close(); //Close input file being read
And this is the State.class.
public class State {
private String stateName;
private String stateCapital;
private String stateAbbrev;
private int statePop;
private String stateRegion;
private int stateRegionNum;
public static final String HEADER_STRING = "\n%-15s %-15s %-4s %-10s %-15s %-8s\n\n";
private static final String DISPLAY_STRING = "%-15s %-15s %-4s %,10d %-15s %1d";
public State(String name, String capital, String abbrev, int pop, String region, int regionNum) {
stateName = name;
stateCapital = capital;
stateAbbrev = abbrev;
statePop = pop;
stateRegion = region;
stateRegionNum = regionNum;
}//End State
public void displayState(int nElems, State myState[]) {
System.out.print(String.format(State.HEADER_STRING, "State", "Capital", "Abbr", "Population", "Region", "Region #"));
for(int x = 0; x < nElems - 1; x++) {
System.out.println(myState[x].toString());
}
}//End displayState
Edit: My attempted Code to output.
String file = "src/program1.states.in.out.txt";
BufferedWriter writer = new BufferedWriter(new FileWriter("src/program1.states.in.out.txt")); {
for(int x = 0; x < Sdriver.getnElems(); x++) {
writer.write(Sdriver.getState(x).toString());
}
}
Created a file, but the file was empty.

You look to be starting in not too bad a spot with this:
String file = "src/program1.states.in.out.txt";
BufferedWriter writer = new BufferedWriter(new FileWriter("src/program1.states.in.out.txt")); {
for(int x = 0; x < Sdriver.getnElems(); x++) {
writer.write(Sdriver.getState(x).toString());
}
}
But consider instead using a PrintWriter object and the displayState() method. Something akin to:
PrintWriter printWriter = null;
try {
String filePath = "src/program1.states.in.out.txt";
writer = new BufferedWriter(new FileWriter(filePath));
PrintWriter printWriter = new PrintWriter(writer);
for(int x = 0; x < Sdriver.getnElems(); x++) {
printWriter.println(Sdriver.getState(x).displayState());
}
} catch (IOException ioE) {
ioE.printStackTrace();
} finally {
if (printWriter != null) {
printWriter.close();
}
}

Scanner from file doesn't seem to be reading file

I'm doing a Phone Directory project and we have to read from a directory file telnos.txt
I'm using a Scanner to load the data from the file telnos.txt, using a loadData method from a previous question I asked here on StackOverflow.
I noticed attempts to find a user always returned Not Found, so I added a few System.out.printlns in the methods to help me see what was going on. It looks like the scanner isn't reading anything from the file. Weirdly, it is printing the name of the file as what should be the first line read, which makes me think I've missed something very very simple here.
Console
run:
telnos.txt
null
loadData tested successfully
Please enter a name to look up: John
-1
Not found
BUILD SUCCESSFUL (total time: 6 seconds)
ArrayPhoneDirectory.java
import java.util.*;
import java.io.*;
public class ArrayPhoneDirectory implements PhoneDirectory {
private static final int INIT_CAPACITY = 100;
private int capacity = INIT_CAPACITY;
// holds telno of directory entries
private int size = 0;
// Array to contain directory entries
private DirectoryEntry[] theDirectory = new DirectoryEntry[capacity];
// Holds name of data file
private final String sourceName = "telnos.txt";
File telnos = new File(sourceName);
// Flag to indicate whether directory was modified since it was last loaded or saved
private boolean modified = false;
// add method stubs as specified in interface to compile
public void loadData(String sourceName) {
Scanner read = new Scanner("telnos.txt").useDelimiter("\\Z");
int i = 1;
String name = null;
String telno = null;
while (read.hasNextLine()) {
if (i % 2 != 0)
name = read.nextLine();
else
telno = read.nextLine();
add(name, telno);
i++;
}
}
public String lookUpEntry(String name) {
int i = find(name);
String a = null;
if (i >= 0) {
a = name + (" is at position " + i + " in the directory");
} else {
a = ("Not found");
}
return a;
}
public String addChangeEntry(String name, String telno) {
for (DirectoryEntry i : theDirectory) {
if (i.getName().equals(name)) {
i.setNumber(telno);
} else {
add(name, telno);
}
}
return null;
}
public String removeEntry(String name) {
for (DirectoryEntry i : theDirectory) {
if (i.getName().equals(name)) {
i.setName(null);
i.setNumber(null);
}
}
return null;
}
public void save() {
PrintWriter writer = null;
// writer = new PrintWriter(FileWriter(sourceName));
}
public String format() {
String a;
a = null;
for (DirectoryEntry i : theDirectory) {
String b;
b = i.getName() + "/n";
String c;
c = i.getNumber() + "/n";
a = a + b + c;
}
return a;
}
// add private methods
// Adds a new entry with the given name and telno to the array of
// directory entries
private void add(String name, String telno) {
System.out.println(name);
System.out.println(telno);
theDirectory[size] = new DirectoryEntry(name, telno);
size = size + 1;
}
// Searches the array of directory entries for a specific name
private int find(String name) {
int result = -1;
for (int count = 0; count < size; count++) {
if (theDirectory[count].getName().equals(name)) {
result = count;
}
System.out.println(result);
}
return result;
}
// Creates a new array of directory entries with twice the capacity
// of the previous one
private void reallocate() {
capacity = capacity * 2;
DirectoryEntry[] newDirectory = new DirectoryEntry[capacity];
System.arraycopy(theDirectory, 0, newDirectory,
0, theDirectory.length);
theDirectory = newDirectory;
}
}
ArrayPhoneDirectoryTester.java
import java.util.Scanner;
public class ArrayPhoneDirectoryTester {
public static void main(String[] args) {
//create a new ArrayPhoneDirectory
PhoneDirectory newTest = new ArrayPhoneDirectory();
newTest.loadData("telnos.txt");
System.out.println("loadData tested successfully");
System.out.print("Please enter a name to look up: ");
Scanner in = new Scanner(System.in);
String name = in.next();
String entryNo = newTest.lookUpEntry(name);
System.out.println(entryNo);
}
}
telnos.txt
John
123
Bill
23
Hello
23455
Frank
12345
Dkddd
31231

In your code:
Scanner read = new Scanner("telnos.txt");
Is not going to load file 'telnos.txt'. It is instead going to create a Scanner object that scans the String "telnos.txt".
To make the Scanner understand that it has to scan a file you have to either:
Scanner read = new Scanner(new File("telnos.txt"));
or create a File object and pass its path to the Scanner constructor.
In case you are getting "File not found" errors you need to check the current working directory. You could run the following lines and see if you are indeed in the right directory in which the file is:
String workingDir = System.getProperty("user.dir");
System.out.println("Current working directory : " + workingDir);
You need to also catch the FileNotFoundException in the function as follows:
public void loadData(String sourceName) {
try {
Scanner read = new Scanner(new File("telnos.txt")).useDelimiter("\\Z");
int i = 1;
String name = null;
String telno = null;
while (read.hasNextLine()) {
if (i % 2 != 0)
name = read.nextLine();
else {
telno = read.nextLine();
add(name, telno);
}
i++;
}
}catch(FileNotFoundException ex) {
System.out.println("File not found:"+ex.getMessage);
}
}

You are actually parsing the filename not the actual file contents.
Instead of:
new Scanner("telnos.txt")
you need
new Scanner( new File( "telnos.txt" ) )
http://docs.oracle.com/javase/7/docs/api/java/util/Scanner.html

How can one get a file's path from an ftp server?

This code gets the file name, but I want to get the file path:
private List <String> checkFiles(FTPClient clients){
List <String> it = new ArrayList <String>();
try {
FTPFile[] ftpFiles = clients.listFiles();
int length = ftpFiles.length;
for (int i = 0; i < length; i++) {
String name = ftpFiles[i].getName();
Calendar date = ftpFiles[i].getTimestamp();
Log.v("aasd", name );
it.add (name);
}
} catch(Exception e) {
e.printStackTrace();
}
return it ;
}

The path is in the client, not the files.
String path = clients.printWorkingDirectory()

if you want specific path
client.changeWorkingDirectory(PathName) eg client.changeWorkingDirectory(folder1/folder2) where folder 2 is inside folder 1
System.out.println(client.printWorkingDirectory)
printWorkingDirectory gives the current path

Below code finds that all files path in any folder on ftp server.
ftpPath is likes that "ftpserver/folder". List contains paths of all files in folder.
public List<string> GetFilesPath(string ftpPath)
{
FtpWebRequest request;
string FtpServerPath = ftpPath;
List<string> filePathList=new List<string>();
try
{
request = WebRequest.Create(new Uri(FtpServerPath)) as FtpWebRequest;
request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
request.UseBinary = true;
request.UsePassive = true;
request.KeepAlive = true;
request.Credentials = new NetworkCredential("ftpuser", "ftpPassword");
request.ConnectionGroupName = "group";
Stream rs = (Stream)request.GetResponse().GetResponseStream();
StreamReader sr = new StreamReader(rs);
string strList = sr.ReadToEnd();
string[] lines = null;
if (strList.Contains("\r\n"))
{
lines = strList.Split(new string[] { "\r\n" }, StringSplitOptions.None);
}
else if (strList.Contains("\n"))
{
lines = strList.Split(new string[] { "\n" }, StringSplitOptions.None);
}
if (lines == null || lines.Length == 0)
return null;
else{
foreach (string line in lines)
{
if (line.Length == 0)
continue;
int x=line.LastIndexOf(' ');
int len = line.Length;
var str = line.Substring( (x+1), (len - x - 1));
var filePath = FtpServerPath+"/"+str;
filePathList.Add(filePath);
}
}
catch (Exception ex)
{
MessageBox.Show("Error: " + ex.Message);
}
}