Iam using lingpipe tool for naive bayes algorithm.I trained it using my trained data and it successfullu tests my test data. But each time I runs the algorithm each time it trains. I don't want to train it each time instead I want to build a model to which I can apply the test data.
public class ClassifyNews {
private static File TRAINING_DIR= new File("train");
private static File TESTING_DIR= new File("test");
private static String[] CATEGORIES
= { "c1",
"c2",
"c3"};
private static int NGRAM_SIZE = 6;
public static void main(String[] args)throws ClassNotFoundException, IOException
{
DynamicLMClassifier<NGramProcessLM> classifier
=DynamicLMClassifier.createNGramProcess(CATEGORIES,NGRAM_SIZE);
for(int i=0; i<CATEGORIES.length; ++i)
{
File classDir = new File(TRAINING_DIR,CATEGORIES[i]);
if (!classDir.isDirectory())
{
String msg = "Could not find training directory="+ classDir
+ "\nTraining directory not found";
System.out.println(msg);
throw new IllegalArgumentException(msg);
}
String[] trainingFiles = classDir.list();
for (int j = 0; j < trainingFiles.length; ++j)
{
File file = new File(classDir,trainingFiles[j]);
String text = Files.readFromFile(file,"ISO-8859-1");
System.out.println("Training on " + CATEGORIES[i] + "/" + trainingFiles[j]);
Classification classification= new Classification(CATEGORIES[i]);
Classified<CharSequence> classified= new Classified<CharSequence>(text,classification);
classifier.handle(classified);}
}
System.out.println("Compiling");
JointClassifier<CharSequence> compiledClassifier
= (JointClassifier<CharSequence>)
AbstractExternalizable.compile(classifier);
boolean storeCategories = true;
JointClassifierEvaluator<CharSequence> evaluator =
new JointClassifierEvaluator
<CharSequence> (compiledClassifier,CATEGORIES,storeCategories);
for(int i = 0; i < CATEGORIES.length; ++i)
{
File classDir = new File(TESTING_DIR,CATEGORIES[i]);
String[] testingFiles = classDir.list();
for (int j=0; j<testingFiles.length; ++j)
{
String text= Files.readFromFile(new File(classDir,testingFiles[j]),"ISO-8859-1");
System.out.print("\nTesting on " + CATEGORIES[i] + "/" + testingFiles[j] + " ");
Classification classification= new Classification(CATEGORIES[i]);
Classified<CharSequence> classified= new Classified<CharSequence>(text,classification);
evaluator.handle(classified);
JointClassification jc =compiledClassifier.classify(text);
String bestCategory = jc.bestCategory();
String details = jc.toString();
System.out.println("\tGot best category of: " + bestCategory);
System.out.println(jc.toString());
}}
}
}
Related
I am new in java. Firstly, I'm sorry for my English. I wrote a code to merge two different txt files in the mergedFile.txt with determined data from the data1.txt and data2.txt. I create 5 different arrays to use them better but I cannot learn the length of words in the textfiles so arrays are using determined parameter. If I want to add another student, this codes don't work. Can you help me?
data1.txt
ID,Name,LastName,Department
12345,John,Samon,Computer Science
14524,David,Souza,Electric and Electronic
.
.
data2.txt
ID,Q1,Q2,Q3,Midterm,Final
12345,100,90,75,89,100
14524,80,70,65,15,90
.
margedFile.txt
ID,Name,Q_Average,Midterm,Final,Department
12345,John,88.3,89,100,Computer Science
14524,David,67.0,100,70,Electric and Electronic
This is ReadData Class
import java.io.FileInputStream;//import java.io library
import java.util.Scanner;//import scanner library
public class ReadData {
public static String[] Read(String filename,String filename2) {
Scanner scan = null;
Scanner scan1 = null;/
FileInputStream input1 = null;
FileInputStream input = null;
String[] result = new String[3];
try {
input = new FileInputStream(filename);
scan = new Scanner(input);
input1 = new FileInputStream(filename2);
scan1 = new Scanner(input1);
String[][] myArray = new String[4][4];
String[][] myArray1 = new String[4][6];
while(scan.hasNext() || scan1.hasNext()) {
for (int i = 0; i < myArray.length; i++) {
String[] split =
scan.nextLine().trim().split(",");
for (int j = 0; j < split.length; j++)
{
myArray[i][j] = split[j];
}
}
for (int i = 0; i < myArray1.length; i++) {
String[] split1 = scan1.nextLine().trim().split(",");
for (int j = 0; j < split1.length; j++) {
myArray1[i][j] = split1[j];
}
}
}
int[][] quiz = new int[3][3];
double[] average = new double[3];
int sum = 0;
double averagee = 0;
for (int i = 0; i < quiz.length; i++) {
for (int j = 0; j < quiz.length; j++) {
quiz[i][j] = Integer.parseInt(myArray1[i+1][j+1]);
sum += quiz[i][j];
}
averagee = sum/quiz.length;
average[i] = averagee;
sum = 0;
}
for (int i = 1; i < myArray1.length; i++) {
for (int j = 1; j < myArray1.length; j++) {
if(myArray[i][0].equalsIgnoreCase(myArray1[j][0])) {
result[i-1] = "\n" + myArray[i][0] + " "+ myArray[i][1] + " " + (average[j-1]) +" "+ myArray1[j][4] +" " + myArray1[j][5] + " "+ myArray[i][3];
//System.out.println(Arrays.deepToString(myArray[i]) + " " + Arrays.deepToString(myArray1[j]));
}
}
}
//System.out.println(Arrays.deepToString(quiz));
//System.out.println(Arrays.toString(average));
}
catch(Exception e) {
System.out.println(e);
}
return result;
}
}
This is WriteData class
import java.io.FileOutputStream;
import java.io.PrintWriter;
import java.util.Arrays;
public class WriteData extends ReadData {
void Write(String filename) {
PrintWriter writer = null;
FileOutputStream output = null;
try {
output = new FileOutputStream(filename);
writer = new PrintWriter(output);
writer.print("ID,Name,Q_Average,Midterm,Final,Department ");
writer.print(Arrays.toString(Read("data1.txt",
"data2.txt")));
writer.flush();
writer.close();
} catch (Exception e) {
// TODO: handle exception
}
}
}
I need to create a setter, so that I can change the name of the merged file as I please. The exportToSingleFile() method takes the arrray input and creates a .csv file for each student and the mergeFile() method reads all of the files and merges the information into 1 output file. I need to rename this output file.
My main method is as follows:
import java.lang.*;
import java.io.*;
public class HW2_Main {
public static void main(String[] args) throws IOException {
myFile studentFiles = new myFile();
studentFiles.exportToSingleFile();
studentFiles.mergeFile();
studentFiles.setFileName("Can't Figure Out");
}
}
This is the new class myFile that I am getting all of my information from to create the objects:
class myFile{
// INPUTS
String[] headers = {"COURSE ID", "TEAM ID", "STUDENT FIRST NAME", "STUDENT LAST NAME",
"STUDENT ID", "ASSIGNMENT ID", "DATE SUBMITTED", "TIME SUBMITTED", "SUBMITTED BY"};
String[][] values = {
{"CMPS280-01", "PELICAN01", "FIRST-1", "LAST-1",
"W1234567", "H01", "08/22/17", "23:53", "W1234567"},
{"CMPS280-01", "PELICAN01", "FIRST-2", "LAST-2",
"W1234566", "H01", "08/22/17", "23:54", "W1234566"},
{"CMPS280-01", "PELICAN01", "FIRST-3", "LAST-3",
"W1234568", "H01", "08/22/17", "20:15", "W1234568"}};
myFile(){
}
void exportToSingleFile() throws IOException{
/*WRITE STUDENT INFO TO TEXT FILES*/
for (int i = 0; i < values.length; i++) {
//Generate file name for each student and prepare for writing to file
PrintWriter printToFile = new PrintWriter(new File("INPUT/"+values[i][1]+"_"+values[i][2]+"_"+values[i][3]+".csv"));
//Write headers to the file
for (int j = 0; j < headers.length - 1; j++) {
printToFile.print(headers[j] + ",");
}
printToFile.println(headers[headers.length - 1]);
//Write values to the file
for (int j = 0; j < headers.length - 1; j++) {
printToFile.print(values[i][j] + ",");
}
printToFile.print(values[i][headers.length - 1]);
printToFile.close();
}//end for
}
void mergeFile() throws IOException{
PrintWriter printToFileAllSubmit = new PrintWriter(new File("OUTPUT/All_Submit.csv"));
//Write headers to the file All_Submit_
for (int j = 0; j < headers.length - 1; j++) {
printToFileAllSubmit.print(headers[j] + ",");
}
printToFileAllSubmit.println(headers[headers.length - 1]);
//Get list of files in folder SUBMIT
File[] fileList = new File("INPUT").listFiles();
for (File f : new File("INPUT").listFiles()){
//System.out.println(f.getName());
BufferedReader bufRead = new BufferedReader(new FileReader(f));
//Read successive lines
while (bufRead.readLine() != null) {
printToFileAllSubmit.println(bufRead.readLine());
}
bufRead.close();
}
printToFileAllSubmit.close();
}
void setFileName(String newFileName){
}
}
I want to implement my input reading method into my main class, I want use my code to parse. It's been fixed now. thanks.
String x;
int count = -1;=
while (str.hasMoreTokens()) {
count++;
x = str.nextToken();
word[count] = x;
System.out.println(count + ": " + word[count]);
}
System.out.println("---Frequency---");
// create unique words
for (int i = 0; i < 7; i++) {
if ((!Arrays.asList(unique).contains(word[i]))) {
unique[i] = word[i];
}
}
// measuring frequency
int[] measure = new int[10];
for (int a = 0; a < 7; a++) {
if (Arrays.asList(unique).contains(word[a])) {
measure[a] += 1;
System.out.println(unique[a] + " : " + measure[a]);
}
}
}
}
private List<String[]> termsDocsArray = new ArrayList<String[]>();
private List<String> allTerms = new ArrayList<String>(); //to hold all terms
private List<double[]> tfidfDocsVector = new ArrayList<double[]>();
/**
To start with your code
String text = "Professor, engineering, data, mining, research";
StringTokenizer str = new StringTokenizer(text);
String word[] = new String[10];
String unique[] = new String[10];
String x;
int count = -1;
while (str.hasMoreTokens()) {
count++;
x = str.nextToken();
word[count] = x;
System.out.println(count + ": " + word[count]);
}
System.out.println("---Frequency---");
// create unique words
for (int i = 0; i < 7; i++) {
if ((!Arrays.asList(unique).contains(word[i]))) {
unique[i] = word[i];
}
}
// measuring frequency
int[] measure = new int[10];
for (int a = 0; a < 7; a++) {
if (Arrays.asList(unique).contains(word[a])) {
measure[a] += 1;
System.out.println(unique[a] + " : " + measure[a]);
}
}
should be in it's own method like .
private void doSomething(){
//This variable will hold all terms of each document in an array.
String text = "Professor, engineering, data, mining, research";
StringTokenizer str = new StringTokenizer(text);
String word[] = new String[10];
String unique[] = new String[10];
String x;
int count = -1;
while (str.hasMoreTokens()) {
count++;
x = str.nextToken();
word[count] = x;
System.out.println(count + ": " + word[count]);
}
System.out.println("---Frequency---");
// create unique words
for (int i = 0; i < 7; i++) {
if ((!Arrays.asList(unique).contains(word[i]))) {
unique[i] = word[i];
}
}
// measuring frequency
int[] measure = new int[10];
for (int a = 0; a < 7; a++) {
if (Arrays.asList(unique).contains(word[a])) {
measure[a] += 1;
System.out.println(unique[a] + " : " + measure[a]);
}
}
}
Secondly in ur given code u have written like
int count = -1;=
which accounts to this error Syntax error on token "=", { expected.It should be
int count = -1;
And since all your code is simply written in class without any method so it is giving you the error saying { expected.
Please make sure you have copied the code correctly.
This is my csv data:
Name,Code,Price,Colour,Type,Stock
A,1001,35000,Red,Car Paint,54
B,1002,56000,Blue,House Paint,90
As you can see, my coding is inefficient.
This is because all the textfields in netbeans do not allow same variable names, I have to give different variable names to each text field (Example: code1, code2, code3, name1, name2,name3)
Can someone help me on how to loop this data so they do it four times and i dont have to repeat the coding? and to skip the process if the fields are blank.
The following is my coding:
try
{
for(int z=0; z<4;z++)
{
String code1;
code1=this.text1.getText();
System.out.println("this is the code: " + code1);
String qty;
int qty1;
qty=this.quantity1.getText();
qty1=Integer.parseInt(qty);
System.out.println("quantity: "+qty1);
String code2;
code2=this.text2.getText();
System.out.println("this is the code: " + code2);
int qty2;
qty=this.quantity2.getText();
qty2=Integer.parseInt(qty);
System.out.println("quantity: "+qty2);
String code3;
code3=this.text3.getText();
System.out.println("this is the code: " + code3);
int qty3;
qty=this.quantity2.getText();
qty3=Integer.parseInt(qty);
System.out.println("quantity: "+qty3);
String code4;
code4=this.text4.getText();
System.out.println("this is the code: " + code4);
int qty4;
qty=this.quantity2.getText();
qty4=Integer.parseInt(qty);
System.out.println("quantity: "+qty4);
int sum=0;
BufferedReader line = new BufferedReader(new FileReader(new File("C:\\Users\\Laura Sutardja\\Documents\\IB DP\\Computer Science HL\\cs\\product.txt")));
String indata;
ArrayList<String[]> dataArr = new ArrayList<>();
String[] club = new String[6];
String[] value;
while ((indata = line.readLine()) != null) {
value = indata.split(",");
dataArr.add(value);
}
for (int i = 0; i < dataArr.size(); i++) {
String[] nameData = dataArr.get(i);
if (nameData[1].equals(code1)) {
System.out.println("Found name.");
name1.setText(""+ nameData[0]);
int price;
price=Integer.parseInt(nameData[2]);
int totalprice=qty1*price;
String total=Integer.toString(totalprice);
price1.setText(total);
sum=sum+totalprice;
break;
}
}
for (int i = 0; i < dataArr.size(); i++) {
String[] nameData = dataArr.get(i);
if (nameData[1].equals(code2)) {
System.out.println("Found name.");
name2.setText(""+ nameData[0]);
int price;
price=Integer.parseInt(nameData[2]);
int totalprice=qty2*price;
String total=Integer.toString(totalprice);
price2.setText(total);
sum=sum+totalprice;
break;
}
}
for (int i = 0; i < dataArr.size(); i++) {
String[] nameData = dataArr.get(i);
if (nameData[1].equals(code3)) {
System.out.println("Found name.");
name3.setText(""+ nameData[0]);
int price;
price=Integer.parseInt(nameData[2]);
int totalprice=qty3*price;
int totalprice3=totalprice;
String total=Integer.toString(totalprice);
price3.setText(total);
sum=sum+totalprice;
break;
}
}
for (int i = 0; i < dataArr.size(); i++) {
String[] nameData = dataArr.get(i);
if (nameData[1].equals(code4)) {
System.out.println("Found name.");
name4.setText(""+ nameData[0]);
int price;
price=Integer.parseInt(nameData[2]);
int totalprice=qty4*price;
int totalprice4=totalprice;
String total=Integer.toString(totalprice);
price4.setText(total);
sum=sum+totalprice;
break;
}
}
total1.setText("Rp. "+sum);
}
}
catch ( IOException iox )
{
System.out.println("Error");
}
Why don't you use a library like http://commons.apache.org/proper/commons-csv/
Solving this problem is actually rather straight forward if you break it down into separate parts.
First you need to solve the problem of loading the data into an internal data representation that is easy to use. Just loading the file into Java is rather simple and you have already done this:
BufferedReader csvFile = new BufferedReader(new FileReader(new File(path)));
String line = "start";
int count = 0;
while((line = csvFile.readLine()) != null){
System.out.println(line);
}
csvFile.close();
The next problem is splitting the line and store it in a meaningful way - for each line.
HashMap<Integer, String> record = new HashMap<Integer, String>();
String[] raw = line.split(",");
for(int i=0;i<raw.length; i++){
record.put(i, raw[i]);
}
Now you state you only want to store records that have non-empty fields so we need to check for that:
HashMap<Integer, String> record = new HashMap<Integer, String>();
String[] raw = line.split(",");
Boolean store = true;
for(int i=0;i<raw.length; i++){
if(raw[i].equals("") || raw[i].equals(null)){
store = false;
break;
}
record.put(i, raw[i]);
}
if(store)
csvData.add(record);
Now, you can load each record of the csv file as a dictionary that you can easily use. All that remains is to save a list of these dictionaries.
ArrayList<Map<Integer, String>> csvData = new ArrayList<Map<Integer, String>>();
BufferedReader csvFile = new BufferedReader(new FileReader(new File(path)));
String line = "start";
int count = 0;
while((line = csvFile.readLine()) != null){
if(count == 0){//skip first line
count++;
continue;
}
HashMap<Integer, String> record = new HashMap<Integer, String>();
String[] raw = line.split(",");
Boolean store = true;
for(int i=0;i<raw.length; i++){
if(raw[i].equals("") || raw[i].equals(null))
{
store = false;
break;
}
record.put(i, raw[i]);
}
if(store)
csvData.add(record);
}
csvFile.close();
Full code snippet that loads in data and easily access whatever information you want:
public class Main {
public static final int NAME = 0;
public static final int CODE = 1;
public static final int PRICE = 2;
public static final int COLOR = 3;
public static final int TYPE = 4;
public static final int STOCK = 5;
public static void main(String[] args) throws IOException{
ArrayList<Map<Integer, String>> csvData = loadCSVFile("C:\\path\\to\\file\\products.txt");
//Print some of the data
System.out.println("---------------------------");
for(Map<Integer, String> record : csvData){
printInfo(record);
}
}
public static ArrayList<Map<Integer, String>> loadCSVFile(String path) throws IOException{
ArrayList<Map<Integer, String>> csvData = new ArrayList<Map<Integer, String>>();
BufferedReader csvFile = new BufferedReader(new FileReader(new File(path)));
String line = "start";
int count = 0;
while((line = csvFile.readLine()) != null){
if(count == 0){
count++;
continue;
}
HashMap<Integer, String> record = new HashMap<Integer, String>();
String[] raw = line.split(",");
Boolean store = true;
for(int i=0;i<raw.length; i++){
if(raw[i].equals("") || raw[i].equals(null))
{
store = false;
break;
}
record.put(i, raw[i]);
}
if(store)
csvData.add(record);
}
csvFile.close();
return csvData;
}
public static void printInfo(Map<Integer, String> record){
System.out.println(record.get(CODE) + " : " + record.get(TYPE));
System.out.println(record.get(NAME) + " : " + record.get(STOCK) + " : " + record.get(PRICE));
System.out.println("---------------------------");
}
}
I have the following little problem... I have this code that uses the method OpenFile() of one class ReadData to read a .txt file and also I have another class ArraysTZones used to create an object that stores 3 arrays (data1,data2,data3) and 3 integers (total1,total2,total3) returned by the method OpenFile(). The problem is that when I try to display each array (data1,data2,data3) using the method getArray() of ArrayTZones it stops and displays the error NullPointerException. Anyone knows how could I fix this?
public static void main (String args[]) throws IOException {
String fileName = ".//data.txt";
int[] def = new int[180];
try {
ReadData file = new ReadData(fileName);
ArraysTZones summaryatz = new ArraysTZones();
summaryatz = file.OpenFile();
for (int i = 0; i < 180; i++)
System.out.print (summaryatz.getArray1()[i] + " ");
System.out.println ("");
System.out.println (summaryatz.getTotal1());
for (int i = 0; i < 180; i++)
System.out.print (summaryatz.getArray2()[i] + " ");
System.out.println ("");
System.out.println (summaryatz.getTotal2());
for (int i = 0; i < 180; i++)
System.out.print (summaryatz.getArray3()[i] + " ");
System.out.println ("");
System.out.println (summaryatz.getTotal3());
}
catch (IOException e) {
System.out.println(e.getMessage());
}
}
Heres OpenFile()
public ArraysTZones OpenFile() throws IOException {
FileReader reader = new FileReader(path);
BufferedReader textReader = new BufferedReader(reader);
int numberOfTimeZones = 3;
int[] data1 = new int[180];
int[] data2 = new int[180];
int[] data3 = new int[180];
int total1 = 0;
int total2 = 0;
int total3 = 0;
ArraysTZones atz = new ArraysTZones();
for (int i = 0; i < numberOfTimeZones; i++){
if (i == 0) {
String firstTimeZone = textReader.readLine();
String[] val = firstTimeZone.split ("\\s+");
for (int u = 0; u < val.length; u++)
{
int stats = (int)(Math.ceil(Math.abs(Double.parseDouble(val[u]))));
total1 += stats;
data1[u] = stats;
}
total1= total1/180;
atz.setTotal1(total1);
atz.setArray1(data1);
}
else
if (i == 1) {
String secondTimeZone = textReader.readLine();
String[] val = secondTimeZone.split ("\\s+");
for (int u = 0; u < val.length; u++)
{
int stats = (int)(Math.ceil(Math.abs(Double.parseDouble(val[u]))));
total2 += stats;
data2[u] = stats;
}
total2= total2/180;
atz.setTotal2(total2);
atz.setArray2(data2);
}
else {
String thirdTimeZone = textReader.readLine();
String[] val = thirdTimeZone.split ("\\s+");
for (int u = 0; u < val.length; u++)
{
int stats = (int)(Math.ceil(Math.abs(Double.parseDouble(val[u]))));
total3 += stats;
data3[u] = stats;
}
total3= total3/180;
atz.setTotal3(total3);
atz.setArray3(data3);
}
}
textReader.close();
return atz;
}
The getArray()
public int[] getArray1 () {
return data1;
}
And setArray()
public void setArray1 (int[] farray) {
int[] data1 = new int[180];
//int[] farray = new int[180];
data1 = farray;
}
The problem seems to be here
public void setArray1 (int[] farray)
{
int[] data1 = new int[180];
//int[] farray = new int[180];
data1 = farray;
}
You're declaring a new variable called data1 and storing the content of farray to it.
After that method is done, that variable will be removed, due to his scope.
Remove int[] from the line int[] data1 = new int[180]; (or just remove the whole line .. it is unnecessary) and your data will be stored in the correct variable that was declared for the class.
public void setArray1 (int[] farray) {
data1 = farray;
}
You have to initialize ArraysTZones