We Keep Coding

Parse many paths in the file tree object. Is there an efficient algorithm?

My code requires creation the file tree of the many file paths as
dir1/file1
dir1/dir2/file2
dir1/dir2/file3
FileTree object visualization example:
dir1
|_file1
|_dir2
|_file2
|_file3
This tree is used for torrent content files visualization in graphical form. It's also used for dynamically show files progress.
In a small number subfolders and files it works effectively, but if paths > 10,000 it takes a lot of memory and time (> 4 seconds and 50 MB RAM).
Is there an efficient algorithm for making such a graph? Most critical for me is the graph make speed.
An example of algorithm implementation can be written in any language, it doesn't matter for me :-)
Thanks in advance.
My Java code for this purpose:
FileTree root = new FileTree(FileTree.ROOT, File.Type.DIR);
FileTree parentTree;
for (String pathToFile : paths) {
parentTree = root;
String[] nodes = FileIOUtils.parsePath(pathToFile); /*String.split(File.separator)*/
for (int i = 0; i < nodes.length; i++) {
/* The last leaf item is a file */
if (i == (nodes.length - 1)) {
parentTree.addChild(new FileTree(nodes[i],
File.Type.FILE, parentTree));
} else {
parentTree.addChild(new FileTree(nodes[i], FileNode.Type.DIR, parentTree));
}
FileTree nextParent = parentTree.getChild(nodes[i]);
/* Skipping leaf nodes */
if (nextParent != null && !nextParent.isFile()) {
parentTree = nextParent;
}
}
}
FileTree class:
public class FileTree {
public static final String ROOT = "/";
/* The name for pointer to the parent node */
public static final String PARENT_DIR = "..";
protected String name;
protected boolean isLeaf;
protected FileTree parent;
protected Map<String, FileTree> children = new LinkedHashMap<>();
public FileTree(String name, int type, FileTree parent) {
this(name, type, parent);
}
public FileTree(String name, int type)
{
this(name, type, null);
}
public FileTree(String name, int type, FileTree parent)
{
this.name = name;
isLeaf = (type == File.Type.FILE);
this.parent = parent;
}
public synchronized void addChild(FileTree node)
{
if (!children.containsKey(node.getName())) {
children.put(node.getName(), node);
}
}
public boolean contains(String name)
{
return children.containsKey(name);
}
public F getChild(String name)
{
return children.get(name);
}
public Collection<FileTree> getChildren()
{
return children.values();
}
public Set<String> getChildrenName()
{
return children.keySet();
}
}
Edit:
It was possible to achieve the speed of creating tree of 1000 subfolders an average of 0.5-1 second (early 30 second).
FileTree root = new BencodeFileTree(FileTree.ROOT, 0L, File.Type.DIR);
FileTree parentTree = root;
/* It allows reduce the number of iterations on the paths with equal beginnings */
String prevPath = "";
/* Sort reduces the returns number to root */
Collections.sort(files);
for (String file : files) {
String path;
/*
* Compare previous path with new path.
* Example:
* prev = dir1/dir2/
* cur = dir1/dir2/file1
* |________|
* equal
*
* prev = dir1/dir2/
* cur = dir3/file2
* |________|
* not equal
*/
if (!prevPath.isEmpty() &&
file.regionMatches(true, 0, prevPath, 0, prevPath.length())) {
/*
* Beginning paths are equal, remove previous path from the new path.
* Example:
* prev = dir1/dir2/
* cur = dir1/dir2/file1
* new = file1
*/
path = file.substring(prevPath.length());
} else {
/* Beginning paths are not equal, return to root */
path = file;
parentTree = root;
}
String[] nodes = FileIOUtils.parsePath(path);
/*
* Remove last node (file) from previous path.
* Example:
* cur = dir1/dir2/file1
* new = dir1/dir2/
*/
prevPath = file.substring(0, file.length() - nodes[nodes.length - 1].length());
/* Iterates path nodes */
for (int i = 0; i < nodes.length; i++) {
if (!parentTree.contains(nodes[i])) {
/* The last leaf item is a file */
parentTree.addChild(makeObject(nodes[i], parentTree,
i == (nodes.length - 1)));
}
FileTree nextParent = parentTree.getChild(nodes[i]);
/* Skipping leaf nodes */
if (!nextParent.isFile()) {
parentTree = nextParent;
}
}
}

The basic algorithm looks good to me, but you are creating a lot of unnecessary FileTree objects when you call addChild that will be immediately thrown away in the (common) case they already exist. You could try passing in the parameters to the constructor and only construct the object if it needs to be inserted:
public synchronized void addChild(String name, int type, FileTree parent)
{
if (!children.containsKey(name)) {
children.put(name, new FileTree(name, type, parent));
}
}
and:
if (i == (nodes.length - 1)) {
parentTree.addChild(nodes[i], File.Type.FILE, parentTree);
} else {
parentTree.addChild(nodes[i], FileNode.Type.DIR, parentTree);
}
It might not be necessary to pass in parentTree: you can just construct it with this.
Another optimization could be to maintain the array of String objects (and associated FileTree nodes) from the previous path that you processed, and scan along until you find an entry that is different to the previous one before adding children.

I would suggest to replace LinkedHashMap with HashMap because first one consumes more memory. The main difference is that HashMap does not guaranty order of iteration over entries. But you could order children in GUI (probably you have some ordering setting anyway). Take a look at this question for some references.
Another suggestion would be to return actual child node from method addChild
public synchronized FileTree addChild(FileTree node) {
return children.putIfAbsent(node.getName(), node);
}
Then inside loop there is no need to call get on map again
FileTree nextParent = parentTree.addChild(...
And there is condition that looks unnecessary
if (nextParent != null && !nextParent.isFile()) {
parentTree = nextParent;
}
Looks like there will be no iteration in loop if current child is a file. So it could be safely replaced with
parentTree = parentTree.addChild(...
After suggestions loop body would look like
for(...) {
int type = if (i == (nodes.length - 1)) ? File.Type.FILE : FileNode.Type.DIR;
parentTree = parentTree.addChild(new FileTree(nodes[i], type, parentTree);
}

Depth First Search Issue

I'm currently learning algorithms and was trying to adapt some code from Algorithms by Robert Sedgewick. Here's a link to the part of the code I'm having problems with: http://algs4.cs.princeton.edu/44sp/DirectedCycle.java.html.
For my code the dependencies are a Vertex class with a constructor that just takes a vertex number. I also have a DirectedEdge class, that has a fromVertex, a toVertex and a weight. Finally, I have an EdgeWeightedDigraph class that has a list of the vertices within the graph and a DirectedEdge adjacency list that holds a list of the edges adjacent to a particular vertex and a test class that just initializes the various instances variables with the data in addition to executing the DirectedCycle.java program. The code compiles and runs but returns a wrong cycle using the data supplied by the book (http://algs4.cs.princeton.edu/42digraph/tinyDG.txt). The book specifies a Directed cycle: 3 5 4 3 but my code returns 3 2 3 instead. I noticed that the code runs fine until it encounters an already marked vertex and the onStack else-if code executes. For some reason the loop that contains the first call to dfs() iterates only once for index = 0 so after a marked vertex is encountered it does not retrace its steps and continue with the next vertex as it should with a depth first search. Not sure what I'm missing but any help will be appreciated. Please let me know if you need me to include the other code for the dependent classes listed above.
Here's my own code:
import java.util.Stack;
public class DirectedCycle {
private boolean[] marked;
private Vertex[] edgeTo;
private Stack<Vertex> cycle;
private boolean[] onStack;
public DirectedCycle(EdgeWeightedDigraph graph) {
onStack = new boolean[graph.getNumOfVertices()];
edgeTo = new Vertex[graph.getNumOfVertices()];
marked = new boolean[graph.getNumOfVertices()];
for (int index = 0; index < graph.getVertices().size(); index++) {
if (!marked[index] && cycle == null) {
dfs(graph, graph.getVertex(index));
}
}
}
private void dfs(EdgeWeightedDigraph graph, Vertex vertex) {
onStack[vertex.getVertexNumber()] = true;
marked[vertex.getVertexNumber()] = true;
for (DirectedEdge w : graph.adjacent(vertex)) {
if (this.hasCycle()) {
return;
}
else if (!marked[w.toVertex().getVertexNumber()]) {
edgeTo[w.toVertex().getVertexNumber()] = vertex;
dfs(graph, w.toVertex());
}
else if (onStack[w.toVertex().getVertexNumber()]) {
cycle = new Stack<>();
for (Vertex v = vertex;
v.getVertexNumber() != w.toVertex().getVertexNumber();
v = edgeTo[v.getVertexNumber()]) {
cycle.push(v);
}
cycle.push(w.toVertex());
cycle.push(vertex);
}
}
onStack[vertex.getVertexNumber()] = false;
}
public boolean hasCycle() {
return cycle != null;
}
public Iterable<Vertex> cycle() {
return cycle;
}
}

OOP in Processing, how should I structure my classes?

Currently I am trying to build an application that will visualize an xml tree in Processing. I am new to Processing (and Java for that matter) and I am having some problems getting my ideas into objects. Currently I have a Node class:
class Node {
//DATA
color textColor;
color boxColor;
float xpos = width/2;
float ypos = 100;
float nodeWidth;
float nodeHeight;
boolean overBox = false;
boolean active = false;
PFont font;
//CONSTRUCTOR
Node(){
textColor = color(0);//sets text color
boxColor = color(244);//sets box color
font = createFont("Gil Sans", 16, true);
textFont(font,50);
nodeWidth = textWidth("Modernism");//INPUT TEXT
nodeHeight = textAscent();//?textDescent()?
rectMode(RADIUS);
}
void displayText(){
fill(textColor);
text("Modernism",xpos-nodeWidth/2,ypos+nodeHeight/2.3);
}
void displayBox(){
//stroke(boxColor);
noStroke();
noFill();
rect(xpos, ypos, nodeWidth/2, nodeHeight/2);
//FOR DEBUGGING OVERBOX
//stroke(135);
//point(300,200);
}
void overBox(){
if(mouseX > xpos-nodeWidth/2 && mouseX < xpos+nodeWidth/2 &&
mouseY > ypos-nodeHeight/2 && mouseY < ypos+nodeHeight/2) {
overBox = true;
}else{
overBox = false;
}
}
void clicked(){
if(active) {
//If box was already clicked, trigger response
textColor = color(0);
overBox = false;
active = false;
}
if(overBox) {
//checks to see if click happened within bounds of box, makes active
textColor = color(100);
active = true;
}
}
boolean activeCheck(){
if(active == true){
return true;
}else{
return false;
}
}
}
And then I want a connection cable drawn between the parent and children of the XML document:
class Connectors{
//DATA
color lineColor;
int lineWeight;
int lineX1 = 12;
int lineY1 = 155;
int lineX2 = 12;
int lineY2 = 475;
//CONSTRUCTOR
Connectors(){
lineColor = color(0);
lineWeight = 2;
}
//FUNCTIONALITY
void displayConnection(){
stroke(lineColor);
strokeWeight(lineWeight);
line(lineX1,lineY1,lineX2,lineY2);
}
}
This is all very rough, but I was wondering how the connections should relate to the nodes. To build the connections, it needs to know where it's parent node is located. Should the connections be a subclass of it's respective node?

You should favor aggregation over inheritance in this case. That is, have the Connector keep a reference to two Nodes.
public class Connector {
// other members
private Node firstNode;
private Node secondNode;
public Connector(Node firstNode, Node secondNode) {
this.firstNode = firstNode;
this.secondNode = secondNode;
}
}
Now the Connector knows about the two Nodes it connects.
The reason why you want to do this is the fact that Nodes and Connectors don't share a lot in common. We have conventions for how we can translate OOP relationships into English.
Use "is a" to describe inheritance.
Use "has a" to describe aggregation and composition.
Which makes more sense? "Connector is a Node," or "Connector has a Node"?
Now, you might want to make an abstract parent class, or even better, an interface, for both of Connector and Node if they do share some things in common, such as the ability to draw them.
public interface MyDrawable {
/**
* Draw this object on the "processing" canvas
* note: 2rs2ts does not know anything about the "processing" library
*/
public void draw();
}
// another file...
public class Connector implements MyDrawable {
// all of the other stuff
public void draw() {
stroke(lineColor);
// etc.
}
}
Inheritance means that you pull everything the parent class does into the child class. (That's the reason for the keyword extends.) If there's stuff about Nodes that is totally separate from stuff about Connectors, don't have one inherit from the other. If you do want to gain OOP abstraction, separate the commonalities into an interface or a common parent class.

I'm assuming you want build your XML parser or tree as a exercise, if not use some java library to walk the xml tree. There are a lot of libraries to do that.
Don't represent connection as a subclass of Node, in no way. A connection is not a node. You can represent your Tree as a special graph, the node is the vertex and the connection is the edges.
You have the option to make the node has other nodes, and that is a connection.
Like this
public class Node {
Node parent;
Node left;
Node right;
// Or if there is N Connections, or as we say in tree leafs
List<Node> connections;
}
Then you use an algorithm to walk the nodes and display the connections starting form the initial node (header), you could use list or hashtables to represent your tree as a collection of nodes. If you study a little of Graph theory or Algorith and Structural Data in computer cience you can do that.

How to use properly the shared BlockingQueue?

I have a binary tree in which every node represents a electronics gate (AND, OR, ...). My mission is to calculate the total value of the tree (like this one in the picture, a binary tree):
This is my code so far (without the threads implementation):
gate_node:
public class gate_node {
gate_node right_c, left_c;
Oprtator op;
int value;
int right_v, left_v;
public gate_node(gate_node right, gate_node left, Oprtator op) {
this.left_c = left;
this.right_c = right;
this.op = op;
right_v = left_v = 0;
}
void add_input(int right_v, int left_v){
this.right_v=right_v;
this.left_v=left_v;
}
int compute(int array_index, int arr_size) {
/*
* The following use of a static sInputCounter assumes that the
* static/global input array is ordered from left to right, irrespective
* of "depth".
*/
final int left, right;
System.out.print(this.op+"(");
if (null != this.left_c) {
left = this.left_c.compute(array_index,arr_size/2);
System.out.print(",");
} else {
left = main_class.arr[array_index];
System.out.print(left + ",");
}
if (null != this.right_c) {
right = this.right_c.compute(array_index + arr_size/2,arr_size/2);
System.out.print(")");
} else {
right = main_class.arr[array_index + 1];
System.out.print(right + ")");
}
return op.calc(left, right);
}
}
Oprtator:
public abstract class Oprtator {
abstract int calc(int x, int y);
}
And
public class and extends Oprtator {
public int calc(int x, int y){
return (x&y);
}
}
Or
public class or extends Oprtator {
public int calc(int x, int y){
return (x|y);
}
}
The tree:
public class tree implements Runnable {
gate_node head;
tree(gate_node head) {
this.head = head;
}
void go_right() {
head = head.right_c;
}
void go_left() {
head = head.left_c;
}
#Override
public void run() {
// TODO Auto-generated method stub
}
}
main class
public class main_class {
public static int arr[] = { 1, 1, 0, 1, 0, 1, 0, 1 };
public static void main(String[] args) {
tree t = new tree(new gate_node(null, null, new and()));
t.head.right_c = new gate_node(null, null, new or());
t.head.right_c.right_c = new gate_node(null, null, new and());
t.head.right_c.left_c = new gate_node(null, null, new and());
t.head.left_c = new gate_node(null, null, new or());
t.head.left_c.right_c = new gate_node(null, null, new and());
t.head.left_c.left_c = new gate_node(null, null, new and());
int res = t.head.compute(0, arr.length);
System.out.println();
System.out.println("The result is: " + res);
}
}
I want to calculate it using thread pool, like this algorithm:
Preparation:
Implement each gate as a class/object. It has to have 2 attributes: input A, input B and a way to calculate result;
Implement a tree. Each node is a pair (gate, next_node). Root is a node with next_node being null. Leaves are nodes such that no other node points to it.
Use a shared (thread safe) queue of nodes. It is initially empty.
There is a fixed number (pick one, does not depend on number of gates) of threads which continuously wait for an element from the queue (unless the result is reached in which case they just quit).
Loop:
Whenever an input occurs on a node put the node in a queue (at the beginning inputs go to leaves). This can be simply implemented by defining add_input method on a gate.
A thread picks up a node from queue:
If one of the input is missing discard it (it will be there one more time when second input appears). Another idea is to put the node in a queue only when both inputs are there.
If both inputs are there, then calculate result and pass it to next_node if it is not null (and put next_node in the queue). If next_node is null, then this is your result - break the loop and finalize.
the only problem is that I don't know how to create a shared BlockingQueue that every node object in the tree can insert himself into it, and how to create an array of fixed sized of threads that constantly waits for new elements in the queue to be available (and then execute them)..... until the head is removed from the list (meaning we are done calculating).
I searched online for BlockingQueue examples but I only found producer and consumer examples and I'm having a hard time to move those example to fit my problem. I would really appreciate it if anyone could try to help me.

I can give you a few starting pointers to get you going :)
To create your threads just spawn that many threads:
for (int i=0;i<MAX_THREADS;i++) {
new Thread(myRunnable).start();
}
You may well want to store a reference to those threads but it isn't required. The threads need no special setup as they are all identical and they all just sit there grabbing items off the queue.
To share a blocking queue the simplest way is just to make it static and final:
static final BlockingQueue blockingQueue();
Now all the threads can access it.
Incidentally if I was doing this I wouldn't use the Queue at all, I'd use a ThreadPoolExecutor and just send the processing to that as new runnables.
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html

Java generic arguments

Going back over my basic ADT stuff here to revise for an interview, and trying to kill two birds with one stone by learning Java while I am. Attempting to write a simple algorithm for a merge sort with a generic linked list ( which I am creating myself). It's proving to be far more difficult than I had first imagined ! Can anyone help me out please ? I will start out working on the basics and will update this post as I get further in.
My code for the generic linked list is as follows :
public class NodeList<T extends Comparable<T> > {
private T head;
public NodeList<T> tail;
public NodeList( T item, NodeList<T> list ) {
head = item;
tail = list;
}
}
I am trying to access this class in another class I have made, which is as follows :
public class MyList<T extends Comparable<T>> {
private NodeList<T> nodes;
private static int size;
public MyList( ) {
nodes = null;
}
public MyList(T[] array ){
for( T item : array ) {
nodes = new NodeList<T>(item, nodes);
}
size = array.length;
}
public void add( T item ) {
nodes = new NodeList<T>( item, nodes );
size++;
}
public void addEnd( T item ) {
NodeList<T> temp = nodes;
while ( temp == null || temp.tail != null) {
temp = temp.tail;
}
size++;
temp.tail = new NodeList<T> ( item, null);
}
I believe, so far, everything to be correct up until the add and addEnd methods, which should add a generic to the start of the list and end of the list respectively.
My code continues with :
public static <S extends Comparable<S>>
MyList<S> sort( MyList<S> list ) {
if ( size > 1 ) {
MyList<S> left = leftHalf( list );
MyList<S> right = rightHalf( list );
list = merge( left, right );
}
return list;
}
private static <S extends Comparable<S>>
MyList<S> merge( MyList<S> left, MyList<S> right ) {
}
private static <S extends Comparable<S>>
MyList<S> leftHalf( MyList<S> list ) {
MyList <S> leftSide = new MyList();
int middle;
if(size % 2 == 1) {
middle = size +1;
} else {
middle = size;
}
for ( int countToMiddle = 0; countToMiddle < middle ; countToMiddle++ ) {
leftSide.addEnd(nodes);
}
// return elements from 0 .. list.size() / 2
}
And I get the error:
addEnd(S) in MyList cannot be applied to (NodeList)
which occurs when I run
leftSide.addEnd(nodes);
Can anyone see a reason for this/ tell me if I am correct up to this point of my work ? Thanks so much again!

If you want NodeList and MyList to only contain Comparable items,
you can replace the generic parameter T with something like:
public class NodeList<T extends Comparable> {
Or
public class NodeList<T extends Comparable<T>> {
And replace where you use Comparable with T. This way, you know T at least implements Comparable's methods.
Oracle's tutorials for generics should be able to help you with getting the hang of them.
One problem you may be having is that you refer to member variables from static functions, like in leftHalf you have:
for ( int countToMiddle = 0; countToMiddle < middle ; countToMiddle++ ) {
leftSide.addEnd(nodes);
}
nodes is a member variable, i.e. a non-static variable, so you can't call it from static methods. For that example, you'd have to get it from the passed MyList:
for ( int countToMiddle = 0; countToMiddle < middle ; countToMiddle++ ) {
leftSide.addEnd(list.nodes);
}
And the same goes for your other static methods that try to use member variables.
Also, the reason you are getting an error like: addEnd(S) in MyList<S> cannot be applied to (NodeList<T>) is because S is, according to your type parameter, a Comparable. NodeList does not extend Comparable!
The two solutions you have is
Make NodeList extend Comparable so you can pass it to MyList.addEnd
Make an overload (i.e. a different method with the same name) for addEnd that takes a NodeList, and add all the items in the passed NodeList to MyList
Or come up with a different solution that better fits the need of your classes.
While I realize you are implementing a linked list just to sharpen your skills for an interview (I wish you good luck!), I just want to add that there is a generified LinkedList already available in Java.

Why do you post almost the same question twice?
You could extend your question, add comments etc.
We already gave you that hint. :)

The error is happening because the class NodeList doesn't have a constructor that receives a generic T class and a NodeList.
Actually, this implementation will replace the reference object that nodes is referring on every loop. You should also fix that.
What you should do is put T to be a Comparable itself, and change the attribute, something like:
public class NodeList<T extends Comparable> {
private T head;
private NodeList tail;
public NodeList( T item, NodeList list ) {
head = item;
tail = list;
}
}
It would be better if you tell us what exactly the code is for.