We Keep Coding

how to export csv table by gephi-toolkit

recently I'm trying to use gephi-toolkit to do some statistics for several networks,it works fine to import the csv file and do some metrics computation, but if I want to export the csv table like what I do below in the Gephi, I do not find any tutorials and there is not description in the javadoc.
since I want to do the same operations for different networks,I want to use gephi-toolkit to implement the same function,but I've checked the demos and google for it but failed to get the result I want.
To describe my question clearer,I post my current code below, now I've tried two methods, one is using the methods of ExporterCSV which only returned me with the matrix in csv file format while what I want is a node csv file and an edge csv file after computing several metrics like betweenness centrality, closeness centrality and so on for each node of each network. And the other method I've tried is using DataTableControllerImpl but it seems that no files has been created, I want to know if there is something wrong with my code, any help is appreciated.
public class Transfer95 {
public void script() {
//Init a project - and therefore a workspace
ProjectController pc = Lookup.getDefault().lookup(ProjectController.class);
pc.newProject();
Workspace workspace = pc.getCurrentWorkspace();
//Get controllers and models
ImportController importController = Lookup.getDefault().lookup(ImportController.class);
//Get models and controllers for this new workspace - will be useful later
GraphModel graphModel = Lookup.getDefault().lookup(GraphController.class).getGraphModel();
//Import file
Container container,container2;
try {
File file_node = new File(getClass().getResource("/resource/season2/club_1_1995.csv").toURI());
container = importController.importFile(file_node);
container.getLoader().setEdgeDefault(EdgeDirectionDefault.DIRECTED); //Force DIRECTED
container.getLoader().setAllowAutoNode(true); //create missing nodes
container.getLoader().setEdgesMergeStrategy(EdgeMergeStrategy.SUM);
container.getLoader().setAutoScale(true);
File file_edge = new File(getClass().getResource("/resource/season2/transfer_1_1995.csv").toURI());
container2 = importController.importFile(file_edge);
container2.getLoader().setEdgeDefault(EdgeDirectionDefault.DIRECTED); //Force DIRECTED
container2.getLoader().setAllowAutoNode(true); //create missing nodes
container2.getLoader().setEdgesMergeStrategy(EdgeMergeStrategy.SUM);
container2.getLoader().setAutoScale(true);
} catch (Exception ex) {
ex.printStackTrace();
return;
}
//Append imported data to GraphAPI
importController.process(container, new DefaultProcessor(), workspace);
importController.process(container2, new AppendProcessor(), workspace); //Use AppendProcessor to append to current workspace
//See if graph is well imported
DirectedGraph graph = graphModel.getDirectedGraph();
System.out.println("Nodes: " + graph.getNodeCount());
System.out.println("Edges: " + graph.getEdgeCount());
//count several metrics
Degree degree=new Degree();
degree.execute(graph.getModel());
System.out.println("Average Degree: "+degree.getAverageDegree());
WeightedDegree weightedDegree=new WeightedDegree();
weightedDegree.execute(graph.getModel());
System.out.println("Average Weighted Degree: "+weightedDegree.getAverageDegree());
ClusteringCoefficient clusteringcoefficient=new ClusteringCoefficient();
clusteringcoefficient.execute(graph.getModel());
System.out.println("Average Clustering Coefficient: "+clusteringcoefficient.getAverageClusteringCoefficient());
GraphDistance graphDistance=new GraphDistance();
graphDistance.execute(graph.getModel());
System.out.println("Average Path Length: "+graphDistance.getPathLength());
System.out.println("Network Diameter: "+graphDistance.getDiameter());
Modularity modularity=new Modularity();
modularity.execute(graph.getModel());
System.out.println("Modularity: "+modularity.getModularity());
GraphDensity graphDensity=new GraphDensity();
graphDensity.execute(graph.getModel());
System.out.println("Graph Density: "+graphDensity.getDensity());
//Export method 1
// ExportController ec = Lookup.getDefault().lookup(ExportController.class);
// ExporterCSV exporterCSV=(ExporterCSV)ec.getExporter("csv");
// try {
//// ec.exportFile(new File("src/resource/output/test_95.csv"));
// ec.exportFile(new File("src/resource/output/test_95.csv"), exporterCSV);
// } catch (IOException ex) {
// ex.printStackTrace();
// return;
// }
//Export method 2
// Lookup.getDefault().lookup(DataTablesController.class).setDataTablesEventListener(DataTableTopComponent.this);
DataTablesControllerImpl csvExp=new DataTablesControllerImpl();
// Lookup.getDefault().lookup(DataTablesControllerImpl.class).setDataTablesEventListener(csvExp.getDataTablesEventListener());
// DataTablesControllerImpl dataTablesController = Lookup.getDefault().lookup(DataTablesController.class);
csvExp.exportCurrentTable();
}
public static void main(String[] args){
Transfer95 test=new Transfer95();
test.script();
}
}

This is not the best solution, but in my case it worked.
Import and use in your project the ExporterSpreadsheet.java class from Gephi source code (GitHub link)
For example, to export nodes table:
try {
ExporterSpreadsheet exporter = new ExporterSpreadsheet();
exporter.setWorkspace(workspace);
exporter.setWriter(new FileWriter(new File("nodes.csv")));
exporter.setTableToExport(ExporterSpreadsheet.ExportTable.NODES);
exporter.execute();
} catch (Exception ex) {
ex.printStackTrace();
}
In the same way, for edges table:
try {
ExporterSpreadsheet exporter = new ExporterSpreadsheet();
exporter.setWorkspace(workspace);
exporter.setWriter(new FileWriter(new File("edges.csv")));
exporter.setTableToExport(ExporterSpreadsheet.ExportTable.EDGES);
exporter.execute();
} catch (Exception ex) {
ex.printStackTrace();
}
As output, you get the same CSV files that Gephi exports using the Export table button from Data Laboratory.

Evaluate the class of a sample using WEKA

I have created a model in Weka using the SMO algorithm. I am trying to evaluate a test sample using the mentioned model to classify it in my two-class problem. I am a bit confused on how to evaluate the sample using Weka Smo code. I have built an empty arff file which contains only the meta-data of the file. I calculate the sample features and I add the vector in arff file. I have created the following function Evaluate in order to evaluate a sample. File template.arff is the template which contains the meta-data of a arff file and models/smo my model.
public static void Evaluate(ArrayList<Float> temp) throws Exception {
temp.add(Float.parseFloat("1"));
System.out.println(temp.size());
double dt[] = new double[temp.size()];
for (int index = 0; index < temp.size(); index++) {
dt[index] = temp.get(index);
}
double data[][] = new double[1][];
data[0] = dt;
weka.classifiers.Classifier c = loadModel(new File("models/"), "/smo"); // loads smo model
File tmp = new File("template.arff"); //loads data template
Instances dataset = new weka.core.converters.ConverterUtils.DataSource(tmp.getAbsolutePath()).getDataSet();
int numInstances = data.length;
for (int inst = 0; inst < numInstances; inst++) {
dataset.add(new Instance(1.0, data[inst]));
}
dataset.setClassIndex(dataset.numAttributes() - 1);
Evaluation eval = new Evaluation(dataset);
//returned evaluated index
double a = eval.evaluateModelOnceAndRecordPrediction(c, dataset.instance(0));
double arr[] = c.distributionForInstance(dataset.instance(0));
System.out.println(" Confidence Scores");
for (int idx = 0; idx < arr.length; idx++) {
System.out.print(arr[idx] + " ");
}
System.out.println();
}
I am not sure if I am right here. I create the sample file. Afterwards I am loading my model. I am wandering if my code is what I need in order to evaluate the class of sample temp. If this code is ok, how can I extract the confidence score and not the binary decision about the class? The structure of template.arff file is:
#relation Dataset
#attribute Attribute0 numeric
#attribute Attribute1 numeric
#attribute Attribute2 numeric
...
#ATTRIBUTE class {1, 2}
#data
Moreover loadModel function is the following:
public static SMO loadModel(File path, String name) throws Exception {
SMO classifier;
FileInputStream fis = new FileInputStream(path + name + ".model");
ObjectInputStream ois = new ObjectInputStream(fis);
classifier = (SMO) ois.readObject();
ois.close();
return classifier;
}
I found this post here which suggest to locate the SMO.java file and change the following line smo.buildClassifier(train, cl1, cl2, true, -1, -1); // from false to true.
However it seems when I did so, I got the same binary output.
My training function:
public void weka_train(File input, String[] options) throws Exception {
long start = System.nanoTime();
File tmp = new File("data.arff");
TwitterTrendSetters obj = new TwitterTrendSetters();
Instances data = new weka.core.converters.ConverterUtils.DataSource(
tmp.getAbsolutePath()).getDataSet();
data.setClassIndex(data.numAttributes() - 1);
Classifier c = null;
String ctype = null;
boolean newmodel = false;
ctype = "SMO";
c = new SMO();
for (int i = 0; i < options.length; i++) {
System.out.print(options[i]);
}
c.setOptions(options);
c.buildClassifier(data);
newmodel = true;
if (newmodel) {
obj.saveModel(c, ctype, new File("models"));
}
}

I have some suggestions but I have no idea whether they will work. Let me know if this works for you.
First use SMO not just the parent object Classifier class. I created a new method loadModelSMO as an example of this.
SMO Class
public static SMO loadModelSMO(File path, String name) throws Exception {
SMO classifier;
FileInputStream fis = new FileInputStream(path + name + ".model");
ObjectInputStream ois = new ObjectInputStream(fis);
classifier = (SMO) ois.readObject();
ois.close();
return classifier;
}
and then
SMO c = loadModelSMO(new File("models/"), "/smo");
...
I found a article that might help you out from the mailing list subject titled
I used SMO with logistic regression but I always get a confidence of 1.0
It suggest to set use the -M to fit your logistics model which can be used through the method
setOptions(java.lang.String[] options)
Also maybe you need to set your build logistics model to true
Confidence score in SMO
c.setBuildLogisticModels(true);
Let me know if this helped at all.

Basically you should try to use the option "-M" for SMO to fit logistic models, in training process. Check the solution proposed here. It should work!

How to read files with an offset from Hadoop using Java

Problem: I want to read a section of a file from HDFS and return it, such as lines 101-120 from a file of 1000 lines.
I don't want to use seek because I have read that it is expensive.
I have log files which I am using PIG to process down into meaningful sets of data. I've been writing an API to return the data for consumption and display by a front end. Those processed data sets can be large enough that I don't want to read the entire file out of Hadoop in one slurp to save wire time and bandwidth. (Let's say 5 - 10MB)
Currently I am using a BufferedReader to return small summary files which is working fine
ArrayList lines = new ArrayList();
...
for (FileStatus item: items) {
// ignoring files like _SUCCESS
if(item.getPath().getName().startsWith("_")) {
continue;
}
in = fs.open(item.getPath());
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String line;
line = br.readLine();
while (line != null) {
line = line.replaceAll("(\\r|\\n)", "");
lines.add(line.split("\t"));
line = br.readLine();
}
}
I've poked around the interwebs quite a bit as well as Stack but haven't found exactly what I need.
Perhaps this is completely the wrong way to go about doing it and I need a completely separate set of code and different functions to manage this. Open to any suggestions.
Thanks!
As added noted based on research from the below discussions:
How does Hadoop process records records split across block boundaries?
Hadoop FileSplit Reading

I think SEEK is a best option for reading files with huge volumes. It did not cause any problems to me as the volume of data that i was reading was in the range of 2 - 3GB. I did not encounter any issues till today but we did use file splitting to handle the large data set. below is the code which you can use for reading purpose and test the same.
public class HDFSClientTesting {
/**
* #param args
*/
public static void main(String[] args) {
// TODO Auto-generated method stub
try{
//System.loadLibrary("libhadoop.so");
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
conf.addResource(new Path("core-site.xml"));
String Filename = "/dir/00000027";
long ByteOffset = 3185041;
SequenceFile.Reader rdr = new SequenceFile.Reader(fs, new Path(Filename), conf);
Text key = new Text();
Text value = new Text();
rdr.seek(ByteOffset);
rdr.next(key,value);
//Plain text
JSONObject jso = new JSONObject(value.toString());
String content = jso.getString("body");
System.out.println("\n\n\n" + content + "\n\n\n");
File file =new File("test.gz");
file.createNewFile();
}
catch (Exception e ){
throw new RuntimeException(e);
}
finally{
}
}
}

How to save an object into a file, so that i can load it later?

I've been trying to save an object, a level in my case, into a file, so that i can later load it again. Basically just a save/load function. I can't get it to work.. I keep getting a file of 5 byte, which seems way to small for what should be in it. I know that it might have something to do with Serializable, but i dont know what. Here is my current code:
(btw, i hard coded the level into the program, because i dont know how to save it properly to a file yet. although that is the goal...)
public class BufferSaveGames {
public void saveGameOutputStream(Level level) throws FileNotFoundException {
ObjectOutputStream output;
try {
output = new ObjectOutputStream((new FileOutputStream("SaveGame.dat")));
output.writeObject(level);
output.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
and the second class Level, which it is loading: (removed code not needed for example)
public class Level implements MouseListener, Serializable {
private Level currentLevel;
private int aantalPoppetjesOpLevel;
private String oplossing;
private int timer;
JPanel[][] levelGrid;
Poppetje[] lijstPoppetjes;
public Level() throws IOException{
levelGrid = new JPanel[5][5];
lijstPoppetjes = new Poppetje[5];
for (int a=0;a<5;a++) {
for (int b=0;b<5;b++) {
levelGrid[a][b] = new JPanel();
levelGrid[a][b].setSize(100,100);
levelGrid[a][b].setVisible(true);
levelGrid[a][b].setLayout(null);
levelGrid[a][b].setOpaque(false);
levelGrid[a][b].addMouseListener(this);
}
}
//bovenste rij
levelGrid[0][0].setLocation(10,10);
levelGrid[0][1].setLocation(112,10);
levelGrid[0][2].setLocation(214,10);
levelGrid[0][3].setLocation(316,10);
levelGrid[0][4].setLocation(418,10);
Poppetje roodPoppetje = new Poppetje("Rood", 4, 4);
Poppetje oranjePoppetje = new Poppetje("Oranje", 0, 4);
Poppetje groenPoppetje = new Poppetje("Groen", 1, 2);
Poppetje paarsPoppetje = new Poppetje("Paars", 2, 1);
Poppetje geelPoppetje = new Poppetje("Geel", 3, 3);
//Poppetje blauwPoppetje = new Poppetje("Blauw");
int tempA = roodPoppetje.getLocatieX(roodPoppetje);
int tempB = roodPoppetje.getLocatieY(roodPoppetje);
levelGrid[tempA][tempB].add(roodPoppetje);
lijstPoppetjes[0] = roodPoppetje;
lijstPoppetjes[0].addMouseListener(this);
tempA = oranjePoppetje.getLocatieX(oranjePoppetje);
tempB = oranjePoppetje.getLocatieY(oranjePoppetje);
levelGrid[tempA][tempB].add(oranjePoppetje);
lijstPoppetjes[1] = oranjePoppetje;
lijstPoppetjes[1].addMouseListener(this);

That could be done with "Serialization/Deserialization". Check out this as a first approach:
http://www.wikihow.com/Serialize-an-Object-in-Java
Also try searching with "serialziation" keyword on your favorite search engine (highly probably google hehe). There are a lot of libraries that do this process in such a high level api. Also there are a lot of libraries that do this with powerful uses; like serialization through databases. Sometimes serialization is done using JSON (as it could be a more universal way to do it). Enjoy with serialization! :-)
edit: also search for JSON, beautiful tool

forgot to put the following line in the OutPutStream code:
Level levelX = new Level();
this solved the problem for the very small save game that it created.

Java: CSV file read & write

I'm reading 2 csv files: store_inventory & new_acquisitions.
I want to be able to compare the store_inventory csv file with new_acquisitions.
1) If the item names match just update the quantity in store_inventory.
2) If new_acquisitions has a new item that does not exist in store_inventory, then add it to the store_inventory.
Here is what i have done so far but its not very good. I added comments where i need to add taks 1 & 2.
Any advice or code to do the above tasks would be great! thanks.
File new_acq = new File("/src/test/new_acquisitions.csv");
Scanner acq_scan = null;
try {
acq_scan = new Scanner(new_acq);
} catch (FileNotFoundException ex) {
Logger.getLogger(mainpage.class.getName()).log(Level.SEVERE, null, ex);
}
String itemName;
int quantity;
Double cost;
Double price;
File store_inv = new File("/src/test/store_inventory.csv");
Scanner invscan = null;
try {
invscan = new Scanner(store_inv);
} catch (FileNotFoundException ex) {
Logger.getLogger(mainpage.class.getName()).log(Level.SEVERE, null, ex);
}
String itemNameInv;
int quantityInv;
Double costInv;
Double priceInv;
while (acq_scan.hasNext()) {
String line = acq_scan.nextLine();
if (line.charAt(0) == '#') {
continue;
}
String[] split = line.split(",");
itemName = split[0];
quantity = Integer.parseInt(split[1]);
cost = Double.parseDouble(split[2]);
price = Double.parseDouble(split[3]);
while(invscan.hasNext()) {
String line2 = invscan.nextLine();
if (line2.charAt(0) == '#') {
continue;
}
String[] split2 = line2.split(",");
itemNameInv = split2[0];
quantityInv = Integer.parseInt(split2[1]);
costInv = Double.parseDouble(split2[2]);
priceInv = Double.parseDouble(split2[3]);
if(itemName == itemNameInv) {
//update quantity
}
}
//add new entry into csv file
}
Thanks again for any help. =]

Suggest you use one of the existing CSV parser such as Commons CSV or Super CSV instead of reinventing the wheel. Should make your life a lot easier.

Your implementation makes the common mistake of breaking the line on commas by using line.split(","). This does not work because the values themselves might have commas in them. If that happens, the value must be quoted, and you need to ignore commas within the quotes. The split method can not do this -- I see this mistake a lot.
Here is the source of an implementation that does it correctly:
http://agiletribe.purplehillsbooks.com/2012/11/23/the-only-class-you-need-for-csv-files/

With help of the open source library uniVocity-parsers, you could develop with pretty clean code as following:
private void processInventory() throws IOException {
/**
* ---------------------------------------------
* Read CSV rows into list of beans you defined
* ---------------------------------------------
*/
// 1st, config the CSV reader with row processor attaching the bean definition
CsvParserSettings settings = new CsvParserSettings();
settings.getFormat().setLineSeparator("\n");
BeanListProcessor<Inventory> rowProcessor = new BeanListProcessor<Inventory>(Inventory.class);
settings.setRowProcessor(rowProcessor);
settings.setHeaderExtractionEnabled(true);
// 2nd, parse all rows from the CSV file into the list of beans you defined
CsvParser parser = new CsvParser(settings);
parser.parse(new FileReader("/src/test/store_inventory.csv"));
List<Inventory> storeInvList = rowProcessor.getBeans();
Iterator<Inventory> storeInvIterator = storeInvList.iterator();
parser.parse(new FileReader("/src/test/new_acquisitions.csv"));
List<Inventory> newAcqList = rowProcessor.getBeans();
Iterator<Inventory> newAcqIterator = newAcqList.iterator();
// 3rd, process the beans with business logic
while (newAcqIterator.hasNext()) {
Inventory newAcq = newAcqIterator.next();
boolean isItemIncluded = false;
while (storeInvIterator.hasNext()) {
Inventory storeInv = storeInvIterator.next();
// 1) If the item names match just update the quantity in store_inventory
if (storeInv.getItemName().equalsIgnoreCase(newAcq.getItemName())) {
storeInv.setQuantity(newAcq.getQuantity());
isItemIncluded = true;
}
}
// 2) If new_acquisitions has a new item that does not exist in store_inventory,
// then add it to the store_inventory.
if (!isItemIncluded) {
storeInvList.add(newAcq);
}
}
}
Just follow this code sample I worked out according to your requirements. Note that the library provided simplified API and significent performance for parsing CSV files.

The operation you are performing will require that for each item in your new acquisitions, you will need to search each item in inventory for a match. This is not only not efficient, but the scanner that you have set up for your inventory file would need to be reset after each item.
I would suggest that you add your new acquisitions and your inventory to collections and then iterate over your new acquisitions and look up the new item in your inventory collection. If the item exists, update the item. If it doesnt, add it to the inventory collection. For this activity, it might be good to write a simple class to contain an inventory item. It could be used for both the new acquisitions and for the inventory. For a fast lookup, I would suggest that you use HashSet or HashMap for your inventory collection.
At the end of the process, dont forget to persist the changes to your inventory file.

As Java doesn’t support parsing of CSV files natively, we have to rely on third party library. Opencsv is one of the best library available for this purpose. It’s open source and is shipped with Apache 2.0 licence which makes it possible for commercial use.
Here, this link should help you and others in the situations!

For writing to CSV
public void writeCSV() {
// Delimiter used in CSV file
private static final String NEW_LINE_SEPARATOR = "\n";
// CSV file header
private static final Object[] FILE_HEADER = { "Empoyee Name","Empoyee Code", "In Time", "Out Time", "Duration", "Is Working Day" };
String fileName = "fileName.csv");
List<Objects> objects = new ArrayList<Objects>();
FileWriter fileWriter = null;
CSVPrinter csvFilePrinter = null;
// Create the CSVFormat object with "\n" as a record delimiter
CSVFormat csvFileFormat = CSVFormat.DEFAULT.withRecordSeparator(NEW_LINE_SEPARATOR);
try {
fileWriter = new FileWriter(fileName);
csvFilePrinter = new CSVPrinter(fileWriter, csvFileFormat);
csvFilePrinter.printRecord(FILE_HEADER);
// Write a new student object list to the CSV file
for (Object object : objects) {
List<String> record = new ArrayList<String>();
record.add(object.getValue1().toString());
record.add(object.getValue2().toString());
record.add(object.getValue3().toString());
csvFilePrinter.printRecord(record);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
fileWriter.flush();
fileWriter.close();
csvFilePrinter.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}

You can use Apache Commons CSV api.
FYI this anwser : https://stackoverflow.com/a/42198895/6549532
Read / Write Example