We Keep Coding

Remove duplicates from array by adding non-duplicates to new array

I need to remove duplicate elements from an array by adding elements that are not repeated in the original array to a new array and output the contents of that.
The problem I am having is that when the new array with no duplicates is printed there are zeros being outputted also.
Thus: does Java fill the array with zeros?
public static boolean hasDuplicates(int arrayNum[])
{
boolean dupFound = false;
int ctr1 =0;
while (ctr1<arrayNum.length && !dupFound)
{
int ctr2 = ctr1+1; // be 1 ahead each time ctr1 increments
while(ctr2<arrayNum.length)
{
if(arrayNum[ctr1] == arrayNum[ctr2])
dupFound = true;
ctr2++;
}
ctr1++;
}
return dupFound;
}
public static int[] removeDuplicates(int[] arrayNum)
{
if(hasDuplicates(arrayNum) == false)
return arrayNum;
else
{
int outArray[] = new int[arrayNum.length];
int ctr1=0;
int ctr2 = ctr1+1;
int index = 0;
boolean dupFound = false;
while(ctr1<arrayNum.length)
{
dupFound = false;
ctr2 = ctr1+1;
while(ctr2<arrayNum.length && !dupFound)
{
if(arrayNum[ctr1] == arrayNum[ctr2])
dupFound = true;
ctr2++;
}
if(dupFound == false)
{
outArray[index] = arrayNum[ctr1];
index++;
}
ctr1++;
}
return outArray;
}
}
public static void testRemoveDuplicates()
{
Scanner input = new Scanner(System.in);
System.out.println("Enter size of input array");
int array[] = new int[input.nextInt()];
System.out.println("Enter number of ints required");
for(int i=0; i<array.length; i++)
{
array[i] = input.nextInt();
}
int outArray[] = new int[array.length];
outArray = removeDuplicates(array);
for(int i=0; i<outArray.length; i++)
{
System.out.println(outArray[i]);
}
}

Here:
int outArray[] = new int[array.length];
That code assumes that you have exactly array.length array elements in your output array. And that is of course a too high number! The point is: when you create a new array of that size, the whole array is initially populated with 0 values. And your code will only "overwrite" a few slots of that output array; and all other slots stay at their initial 0 default value.
The point is: you first have to compute how many duplicates you have in your input array; and then you create a new array with exactly that number.
Alternatively, you could be using List<Integer> instead of int[]; as Java collections have the ability to grow dynamically. (or, you can keep increasing that array for collecting duplicates "manually"; that can be done, too - just a bit of complicated code to get there).
And the direct immediate answer is: yes, exactly - Java arrays are "pre-filled"; see here.

You can fix all of your problems (and probably make the code substantially faster in the process since it'll no longer have quadratic complexity) by using standard Java API calls to store the unique elements in a Set, and then turn the values in that set back into an array.
The main caveat is that you'd need to use Integer rather than int:
Set<Integer> s = new LinkedHashSet<Integer>(Arrays.asList(inputArray));
Integer[] outputArray = s.toArray(new Integer[0]);
The LinkedHashSet preserves insertion order, so you'll get the elements back in the same order as they originally appeared albeit without the duplicates.

May be you are wasting too much memory here by considering the worst case scenario where all input given are different while declaring the output array size. More Optimized Approach is that you can have an List<Integer> or ArrayList<Integer> which can store the unique values and then at last , If you want unique values in Array than declare the ArraySize same as the ArrayList<Integer> or List<Integer> size and just in one linear loop , You can copy all the data.
Does Java fill the array with zeros?
Yes , Whenever you declare the integer array , It will be having all its elements as 0. If you want to reset the value of Array to a particular value , You can use this method Arrays.fill(int array[] , int default_value). This can be done in O(list_size) complexity which is linear.
For your purpose more better approach would be use of HashSet<Integer> which holds only unique elements and which is Dynamic in nature so no need to waste extra space as well as it can make your work very easy.
So first you need to all elements in the HashSet<Integer> and then by using Iterator you can easily iterate through it but the insertion order can be disturbed here. So as replied by #Alnitak You can use LinkedHashSet<Integer> to have insertion order same.
A sample code using HashSet :
HashSet<Integer> set=new HashSet<Integer>();
int total_inputs=sc.nextInt();
for(int i=0;i<total_inputs;i++)
set.add(sc.nextInt());
Iterator itr=set.iterator();
while(itr.hasNext())
System.out.println((int)itr.next());
Note : Here input order will not be preserved.

Does Java fill the array with zeros?
Yes, java will initialize every element of your int array to 0. Therefore using arr.length will not work for you as it doesn't return the logical length of your array. You need to track the number of elements of your array yourself. The following line in your code has no meaning
int outArray[] = new int[array.length];
because outArray starts pointing to a different array as returned by your method removeDuplicates.
I need to remove duplicate elements from an array by adding elements
that are not repeated in the original array to a new array
The easiest thing you can do is the following way:
In your method removeDuplicates,
Find the highest number from the given input array, increment it by 1 and assign it to a variable.
int max = Arrays.stream(arrayNum).max().getAsInt() + 1;
Modify your if block from
if(arrayNum[ctr1] == arrayNum[ctr2])
dupFound = true;
to
if(arrayNum[ctr1] == arrayNum[ctr2]) {
dupFound = true;
arrayNum[ctrl] = max;
max++;
}

Java - Return random index of specific character in string

So given a string such as: 0100101, I want to return a random single index of one of the positions of a 1 (1, 5, 6).
So far I'm using:
protected int getRandomBirthIndex(String s) {
ArrayList<Integer> birthIndicies = new ArrayList<Integer>();
for (int i = 0; i < s.length(); i++) {
if ((s.charAt(i) == '1')) {
birthIndicies.add(i);
}
}
return birthIndicies.get(Randomizer.nextInt(birthIndicies.size()));
}
However, it's causing a bottle-neck on my code (45% of CPU time is in this method), as the strings are over 4000 characters long. Can anyone think of a more efficient way to do this?

If you're interested in a single index of one of the positions with 1, and assuming there is at least one 1 in your input, you can just do this:
String input = "0100101";
final int n=input.length();
Random generator = new Random();
char c=0;
int i=0;
do{
i = generator.nextInt(n);
c=input.charAt(i);
}while(c!='1');
System.out.println(i);
This solution is fast and does not consume much memory, for example when 1 and 0 are distributed uniformly. As highlighted by #paxdiablo it can perform poorly in some cases, for example when 1 are scarce.

You could use String.indexOf(int) to find each 1 (instead of iterating every character). I would also prefer to program to the List interface and to use the diamond operator <>. Something like,
private static Random rand = new Random();
protected int getRandomBirthIndex(String s) {
List<Integer> birthIndicies = new ArrayList<>();
int index = s.indexOf('1');
while (index > -1) {
birthIndicies.add(index);
index = s.indexOf('1', index + 1);
}
return birthIndicies.get(rand.nextInt(birthIndicies.size()));
}
Finally, if you need to do this many times, save the List as a field and re-use it (instead of calculating the indices every time). For example with memoization,
private static Random rand = new Random();
private static Map<String, List<Integer>> memo = new HashMap<>();
protected int getRandomBirthIndex(String s) {
List<Integer> birthIndicies;
if (!memo.containsKey(s)) {
birthIndicies = new ArrayList<>();
int index = s.indexOf('1');
while (index > -1) {
birthIndicies.add(index);
index = s.indexOf('1', index + 1);
}
memo.put(s, birthIndicies);
} else {
birthIndicies = memo.get(s);
}
return birthIndicies.get(rand.nextInt(birthIndicies.size()));
}

Well, one way would be to remove the creation of the list each time, by caching the list based on the string itself, assuming the strings are used more often than they're changed. If they're not, then caching methods won't help.
The caching method involves, rather than having just a string, have an object consisting of:
current string;
cached string; and
list based on the cached string.
You can provide a function to the clients to create such an object from a given string and it would set the string and the cached string to whatever was passed in, then calculate the list. Another function would be used to change the current string to something else.
The getRandomBirthIndex() function then receives this structure (rather than the string) and follows the rule set:
if the current and cached strings are different, set the cached string to be the same as the current string, then recalculate the list based on that.
in any case, return a random element from the list.
That way, if the list changes rarely, you avoid the expensive recalculation where it's not necessary.
In pseudo-code, something like this should suffice:
# Constructs fastie from string.
# Sets cached string to something other than
# that passed in (lazy list creation).
def fastie.constructor(string s):
me.current = s
me.cached = s + "!"
# Changes current string in fastie. No list update in
# case you change it again before needing an element.
def fastie.changeString(string s):
me.current = s
# Get a random index, will recalculate list first but
# only if necessary. Empty list returns index of -1.
def fastie.getRandomBirthIndex()
me.recalcListFromCached()
if me.list.size() == 0:
return -1
return me.list[random(me.list.size())]
# Recalculates the list from the current string.
# Done on an as-needed basis.
def fastie.recalcListFromCached():
if me.current != me.cached:
me.cached = me.current
me.list = empty
for idx = 0 to me.cached.length() - 1 inclusive:
if me.cached[idx] == '1':
me.list.append(idx)
You also have the option of speeding up the actual searching for the 1 character by, for example, useing indexOf() to locate them using the underlying Java libraries rather than checking each character individually in your own code (again, pseudo-code):
def fastie.recalcListFromCached():
if me.current != me.cached:
me.cached = me.current
me.list = empty
idx = me.cached.indexOf('1')
while idx != -1:
me.list.append(idx)
idx = me.cached.indexOf('1', idx + 1)
This method can be used even if you don't cache the values. It's likely to be faster using Java's probably-optimised string search code than doing it yourself.
However, you should keep in mind that your supposed problem of spending 45% of time in that code may not be an issue at all. It's not so much the proportion of time spent there as it is the absolute amount of time.
By that, I mean it probably makes no difference what percentage of the time being spent in that function if it finishes in 0.001 seconds (and you're not wanting to process thousands of strings per second). You should only really become concerned if the effects become noticeable to the user of your software somehow. Otherwise, optimisation is pretty much wasted effort.

You can even try this with best case complexity O(1) and in worst case it might go to O(n) or purely worst case can be infinity as it purely depends on Randomizer function that you are using.
private static Random rand = new Random();
protected int getRandomBirthIndex(String s) {
List<Integer> birthIndicies = new ArrayList<>();
int index = s.indexOf('1');
while (index > -1) {
birthIndicies.add(index);
index = s.indexOf('1', index + 1);
}
return birthIndicies.get(rand.nextInt(birthIndicies.size()));
}

If your Strings are very long and you're sure it contains a lot of 1s (or the String you're looking for), its probably faster to randomly "poke around" in the String until you find what you are looking for. So you save the time iterating the String:
String s = "0100101";
int index = ThreadLocalRandom.current().nextInt(s.length());
while(s.charAt(index) != '1') {
System.out.println("got not a 1, trying again");
index = ThreadLocalRandom.current().nextInt(s.length());
}
System.out.println("found: " + index + " - " + s.charAt(index));
I'm not sure about the statistics, but it rare cases might happen that this Solution take much longer that the iterating solution. On case is a long String with only a very few occurrences of the search string.
If the Source-String doesn't contain the search String at all, this code will run forever!

One possibility is to use a short-circuited Fisher-Yates style shuffle. Create an array of the indices and start shuffling it. As soon as the next shuffled element points to a one, return that index. If you find you've iterated through indices without finding a one, then this string contains only zeros so return -1.
If the length of the strings is always the same, the array indices can be static as shown below, and doesn't need reinitializing on new invocations. If not, you'll have to move the declaration of indices into the method and initialize it each time with the correct index set. The code below was written for strings of length 7, such as your example of 0100101.
// delete this and uncomment below if string lengths vary
private static int[] indices = { 0, 1, 2, 3, 4, 5, 6 };
protected int getRandomBirthIndex(String s) {
int tmp;
/*
* int[] indices = new int[s.length()];
* for (int i = 0; i < s.length(); ++i) indices[i] = i;
*/
for (int i = 0; i < s.length(); i++) {
int j = randomizer.nextInt(indices.length - i) + i;
if (j != i) { // swap to shuffle
tmp = indices[i];
indices[i] = indices[j];
indices[j] = tmp;
}
if ((s.charAt(indices[i]) == '1')) {
return indices[i];
}
}
return -1;
}
This approach terminates quickly if 1's are dense, guarantees termination after s.length() iterations even if there aren't any 1's, and the locations returned are uniform across the set of 1's.

checking if my array elements meet requirements

I need to create a method which checks each element in my array to see if it is true or false, each element holds several values such as mass, formula, area etc for one compound, and in total there are 30 compounds (so the array has 30 elements). I need an algorithm to ask if mass < 50 and area > 5 = true .
My properties class looks like:
public void addProperty (Properties pro )
{
if (listSize >=listlength)
{
listlength = 2 * listlength;
TheProperties [] newList = new TheProperties [listlength];
System.arraycopy (proList, 0, newList, 0, proList.length);
proList = newList;
}
//add new property object in the next position
proList[listSize] = pro;
listSize++;
}
public int getSize()
{
return listSize;
}
//returns properties at a paticular position in list numbered from 0
public TheProperties getProperties (int pos)
{
return proList[pos];
}
}
and after using my getters/setters from TheProperties I put all the information in the array using the following;
TheProperties tp = new properties();
string i = tp.getMass();
String y = tp.getArea();
//etc
theList.addProperty(tp);
I then used the following to save an output of the file;
StringBuilder builder = new StringBuilder();
for (int i=0; i<theList.getSize(); i++)
{
if(theList.getProperties(i).getFormatted() != null)
{
builder.append(theList.getProperties(i).getFormatted());
builder.append("\n");
}
}
SaveFile sf = new SaveFile(this, builder.toString());
I just cant work out how to interrogate each compound individually for whether they reach the value or not, reading a file in and having a value for each one which then gets saved has worked, and I can write an if statement for the requirements to check against, but how to actually check the elements for each compound match the requirements? I am trying to word this best I can, I am still working on my fairly poor java skills.

Not entirely sure what you are after, I found your description quite hard to understand, but if you want to see if the mass is less than 50 and the area is greater than 5, a simple if statement, like so, will do.
if (tp.getMass() < 50 && tp.getArea() > 5) {}
Although, you will again, have to instantiate tp and ensure it has been given its attributes through some sort of constructor.

Lots of ways to do this, which makes it hard to answer.
You could check at creation time, and just not even add the invalid ones to the list. That would mean you only have to loop once.
If you just want to save the output to the file, and not do anything else, I suggest you combine the reading and writing into one function.
Open up the read and the write file
while(read from file){
check value is ok
write to file
}
close both files
The advantage of doing it this way are:
You only loop through once, not three times, so it is faster
You never have to store the whole list in memory, so you can handle really large files, with thousands of elements.

In case the requirements changes, you can write method that uses Predicate<T>, which is a FunctionalInterface designed for such cases (functionalInterfaces was introduced in Java 8):
// check each element of the list by custom condition (predicate)
public static void checkProperties(TheList list, Predicate<TheProperties> criteria) {
for (int i=0; i < list.getSize(); i++) {
TheProperties tp = list.get(i);
if (!criteria.apply(tp)) {
throw new IllegalArgumentException(
"TheProperty at index " + i + " does not meet the specified criteria");
}
}
}
If you want to check if mass < 50 and area > 5, you would write:
checkProperties(theList, new Predicate<TheProperties> () {
#Override
public boolean apply(TheProperties tp) {
return tp.getMass() < 50 && tp.getArea() > 5;
}
}
This can be shortened by using lambda expression:
checkProperties(theList, (TheProperties tp) -> {
return tp.getMass() < 50 && tp.getArea() > 5;
});

Optimal render draw-order function with specified z-index values

I found recently the default renderable sort function in LibGDX wasn't quite up to my needs. (see; Draw order changes strangely as camera moves? )
Essentially a few objects rendered in front when they should render behind.
Fortunately, the renderables in question always have a guarantied relationship. The objects are attached to eachother so when one moves the other moves. One object can be seen as being literally "pinned" to the other, so always in front.
This gave me the idea that if I specified a "z-index" (int) and "groupname" (String) for each object, I could manually take over the draw order, and for things with the same groupname, ensure they are positioned next to eachother in the list, in the order specified by the z-index. (low to high)
//For example an array of renderables like
0."testgroup2",11
1."testgroup",20
2."testgroup2",10
3.(no zindex attribute)
4."testgroup",50
//Should sort to become
0."testgroup",20
1."testgroup",50
2.(no zindex attribute)
3."testgroup2",10
4."testgroup2",11
// assuming the object2 in testgroup2 are closer to the camera, the one without a index second closest, and the rest furthest<br>
//(It is assumed that things within the same group wont be drastically different distances)
I implemented a sort system in libgdx to do this as followed;
/**
* The goal of this sorter is to sort the renderables the same way LibGDX would do normally (in DefaultRenderableSorter)<br>
* except if they have a ZIndex Attribute.<br>
* A Zindex attribute provides a groupname string and a number.<br>
* Renderables with the attribute are placed next to others of the same group, with the order within the group determined by the number<br>
*
* For example an array of renderables like;<br><br>
* 0."testgroup",20<br>
* 1."testgroup2",10<br>
* 2.(no zindex attribute)<br>
* 3."testgroup",50<br>
* <br>Should become;<br><br>
* 0."testgroup",20<br>
* 1."testgroup",50<br>
* 2.(no zindex attribute)<br>
* 3."testgroup2",10<br>
* <br>
* assuming the object in testgroup2 is closer to the camera, the one without a index second closest, and the rest furthest<br>
* (It is assumed that things within the same group wont be drastically different distances)<br>
*
* #param camera - the camera in use to determine normal sort order when we cant place in a existing group
* #param resultList - an array of renderables to change the order of
*/
private void customSorter(Camera camera, Array<Renderable> resultList) {
//make a copy of the list to sort. (This is probably a bad start)
Array <Renderable> renderables = new Array <Renderable> (resultList);
//we work by clearing and rebuilding the Renderables array (probably not a good method)
resultList.clear();
//loop over the copy we made
for (Renderable o1 : renderables) {
//depending of if the Renderable as a ZIndexAttribute or not, we sort it differently
//if it has one we do the following....
if (o1.material.has(ZIndexAttribute.ID)){
//get the index and index group name of it.
int o1Index = ((ZIndexAttribute)o1.material.get(ZIndexAttribute.ID)).zIndex;
String o1GroupName = ((ZIndexAttribute)o1.material.get(ZIndexAttribute.ID)).group;
//setup some variables
boolean placementFound = false; //Determines if a placement was found for this renderable (this happens if it comes across another with the same groupname)
int defaultPosition = -1; //if it doesn't find another renderable with the same groupname, this will be its position in the list. Consider this the "natural" position based on distance from camera
//start looping over all objects so far in the results (urg, told you this was probably not a good method)
for (int i = 0; i < resultList.size; i++) {
//first get the renderable and its ZIndexAttribute (null if none found)
Renderable o2 = resultList.get(i);
ZIndexAttribute o2szindex = ((ZIndexAttribute)o2.material.get(ZIndexAttribute.ID));
if (o2szindex!=null){
//if the renderable we are comparing too has a zindex, then we get its information
int o2index = o2szindex.zIndex;
String o2groupname = o2szindex.group;
//if its in the same group as o1, then we start the processing of placing them nexto eachother
if (o2groupname.equals(o1GroupName)){
//we either place it in front or behind based on zindex
if (o1Index<o2index){
//if lower z-index then behind it
resultList.insert(i, o1);
placementFound = true;
break;
}
if (o1Index>o2index){
//if higher z-index then it should go in front UNLESS there is another of this group already there too
//in which case we just continue (which will cause this to fire again on the next renderable in the inner loop)
if (resultList.size>(i+1)){
Renderable o3 = resultList.get(i+1);
ZIndexAttribute o3szindex = ((ZIndexAttribute)o3.material.get(ZIndexAttribute.ID));
if (o3szindex!=null){
String o3groupname = o3szindex.group;
if (o3groupname!=null && o3groupname.equals(o1GroupName)){
//the next element is also a renderable with the same groupname, so we loop and test that one instead
continue;
}
}
}
// Gdx.app.log("zindex", "__..placeing at:"+(i+1));
//else we place after the current one
resultList.insert(i+1, o1);
placementFound = true;
break;
}
}
}
//if no matching groupname found we need to work out a default placement.
int placement = normalcompare(o1, o2); //normal compare is the compare function in DefaultRenderableSorter.
if (placement>0){
//after then we skip
//(we are waiting till we are either under something or at the end
} else {
//if placement is before, then we remember this position as the default (but keep looking as there still might be matching groupname, which should take priority)
defaultPosition = i;
//break; //break out the loop
}
}
//if we have checked all the renderables positioned in the results list, and none were found with matching groupname
//then we use the defaultposition to insert it
if (!placementFound){
//Gdx.app.log("zindex", "__no placement found using default which is:"+defaultPosition);
if (defaultPosition>-1){
resultList.insert(defaultPosition, o1);
} else {
resultList.add(o1);
}
}
continue;
}
//...(breath out)...
//ok NOW we do placement for things that have no got a ZIndexSpecified
boolean placementFound = false;
//again, loop over all the elements in results
for (int i = 0; i < resultList.size; i++) {
Renderable o2 = resultList.get(i);
//if not we compare by default to place before/after
int placement = normalcompare(o1, o2);
if (placement>0){
//after then we skip
//(we are waiting till we are either under something or at the end)
continue;
} else {
//before
resultList.insert(i, o1);
placementFound = true;
break; //break out the loop
}
}
//if no placement found we go at the end by default
if (!placementFound){
resultList.add(o1);
};
} //go back to check the next element in the incomeing list of renderables (that is, the copy we made at the start)
//done
}
//Copy of the default sorters compare function
//;
private Camera camera;
private final Vector3 tmpV1 = new Vector3();
private final Vector3 tmpV2 = new Vector3();
public int normalcompare (final Renderable o1, final Renderable o2) {
final boolean b1 = o1.material.has(BlendingAttribute.Type) && ((BlendingAttribute)o1.material.get(BlendingAttribute.Type)).blended;
final boolean b2 = o2.material.has(BlendingAttribute.Type) && ((BlendingAttribute)o2.material.get(BlendingAttribute.Type)).blended;
if (b1 != b2) return b1 ? 1 : -1;
// FIXME implement better sorting algorithm
// final boolean same = o1.shader == o2.shader && o1.mesh == o2.mesh && (o1.lights == null) == (o2.lights == null) &&
// o1.material.equals(o2.material);
o1.worldTransform.getTranslation(tmpV1);
o2.worldTransform.getTranslation(tmpV2);
final float dst = (int)(1000f * camera.position.dst2(tmpV1)) - (int)(1000f * camera.position.dst2(tmpV2));
final int result = dst < 0 ? -1 : (dst > 0 ? 1 : 0);
return b1 ? -result : result;
}
As far as I can tell my customSorter function produces the order I want - the renderables now look like they are drawn in the right order.
However, this also seems like a hackjob, and I am sure my sorting algorithm is horrendously inefficient.
I would like advice on how to either;
a) Improve my own algorithm, especially in regards to any quirks to bare in mind when doing cross-platform LibGDX development (ie, array types, memory management in regards to android/web etc)
b) Alternative more efficient solutions having a similar "z index override" of the normal draw-order sorting.
Notes;
. The grouping is necessary. This is because while things are firmly stuck relatively to eachother within a group, groups themselves can also move about in front/behind eachother. (but not between). This makes it tricky to do a "global" override of the draw order, rather then a local one per group.
. If it helps, I can add/change the zindexattribute object in any way.
. I am thinking somehow "pre-storeing" each group of objects in a array could help things, but not 100% sure how.

First of all do never copy a list if not needed. The list with renderables could be really huge since it also could contain resources. Copying will be very very slow. If you need something local and you need performance try to make it final since it can improve the performance.
So a simple approach would be the default sorting of Java. You need to implement a Comperator for your class for example the Class with z index could look like this:
public class MyRenderable {
private float z_index;
public MyRenderable(float i)
{
z_index = i;
}
public float getZ_index() {
return z_index;
}
public void setZ_index(float z_index) {
this.z_index = z_index;
}
}
If you want a faster sort since your list wont change that much on runtime you could implement a insertion sort since it does a faster job if the list is kind of presorted. If it is not pre sorted it does take longer but in general it should only be the first sort call where it is alot disordered in your case.
private void sortList(ArrayList<MyRenderable> array) {
// double starttime = System.nanoTime();
for (int i = 1; i < array.size(); i++) {
final MyRenderable temp = array.get(i);
int j = i - 1;
while (j >= 0 && array.get(j).getZ_index() < temp.getZ_index()) {
array.set(j + 1, array.get(j));
j--;
}
array.set(j + 1, temp);
}
// System.out.println("Time taken: " + (System.nanoTime() - starttime));
}
To use this method you simply call it with your Array
sortList(renderbales);
In your case you need to take care of the ones that do not have a Z index. Maybe you could give them a 0 since they'll get sorted at the right position(i guess). Else you can use the given methods in z case and the regular in no z case as you do already.
After the conversation in the comments. I dont think it is a good idea to push everything into one list. It's hard to sort and would be very slow. A better approach would be a list of groups. Since you want to have groups, programm a group. Do not use String names, use IDs or types (way more easy to sort and it doesn't really matter). So a simple group would be this:
public class Group{
//think about privates and getters or methods to add things which also checks some conditions and so on
public int groupType;
public ArrayList<MyRenderable> renderables;
}
And now all your groups into a list. (this contains all your renderbales then)
ArrayList<Group> allRenderables = new ArrayList<>();
Last but not least sort the groups and sort the renderables. Since i dont think that your group ids/names will change on runtime, sort them once or even use a SortedSet instead of a ArrayList. But basically the whole sorting looks like this:
for(Group g: allRenderables)
sortRenderables(g.renderables); //now every group is sorted
//now sort by group names
sortGroup(allRenderables);
With the following insertionsorts as shown above
public static void sortRenderables(ArrayList<MyRenderable> array) {
for (int i = 1; i < array.size(); i++) {
final MyRenderable temp = array.get(i);
int j = i - 1;
while (j >= 0 && array.get(j).getZ_index() < temp.getZ_index()) {
array.set(j + 1, array.get(j));
j--;
}
array.set(j + 1, temp);
}
}
public static void sortGroup(ArrayList<Group> array) {
for (int i = 1; i < array.size(); i++) {
final Group temp = array.get(i);
int j = i - 1;
while (j >= 0 && array.get(j).groupType < temp.groupType) {
array.set(j + 1, array.get(j));
j--;
}
array.set(j + 1, temp);
}
}

Java : linear algorithm but non-linear performance drop, where does it come from?

I am currently having heavy performance issues with an application I'm developping in natural language processing. Basically, given texts, it gathers various data and does a bit of number crunching.
And for every sentence, it does EXACTLY the same. The algorithms applied to gather the statistics do not evolve with previously read data and therefore stay the same.
The issue is that the processing time does not evolve linearly at all: 1 min for 10k sentences, 1 hour for 100k and days for 1M...
I tried everything I could, from re-implementing basic data structures to object pooling to recycles instances. The behavior doesn't change. I get non-linear increase in time that seem impossible to justify by a little more hashmap collisions, nor by IO waiting, nor by anything! Java starts to be sluggish when data increases and I feel totally helpless.
If you want an example, just try the following: count the number of occurences of each word in a big file. Some code is shown below. By doing this, it takes me 3 seconds over 100k sentences and 326 seconds over 1.6M ...so a multiplicator of 110 times instead of 16 times. As data grows more, it just get worse...
Here is a code sample:
Note that I compare strings by reference (for efficiency reasons), this can be done thanks to the 'String.intern()' method which returns a unique reference per string. And the map is never re-hashed during the whole process for the numbers given above.
public class DataGathering
{
SimpleRefCounter<String> counts = new SimpleRefCounter<String>(1000000);
private void makeCounts(String path) throws IOException
{
BufferedReader file_src = new BufferedReader(new FileReader(path));
String line_src;
int n = 0;
while (file_src.ready())
{
n++;
if (n % 10000 == 0)
System.out.print(".");
if (n % 100000 == 0)
System.out.println("");
line_src = file_src.readLine();
String[] src_tokens = line_src.split("[ ,.;:?!'\"]");
for (int i = 0; i < src_tokens.length; i++)
{
String src = src_tokens[i].intern();
counts.bump(src);
}
}
file_src.close();
}
public static void main(String[] args) throws IOException
{
String path = "some_big_file.txt";
long timestamp = System.currentTimeMillis();
DataGathering dg = new DataGathering();
dg.makeCounts(path);
long time = (System.currentTimeMillis() - timestamp) / 1000;
System.out.println("\nElapsed time: " + time + "s.");
}
}
public class SimpleRefCounter<K>
{
static final double GROW_FACTOR = 2;
static final double LOAD_FACTOR = 0.5;
private int capacity;
private Object[] keys;
private int[] counts;
public SimpleRefCounter()
{
this(1000);
}
public SimpleRefCounter(int capacity)
{
this.capacity = capacity;
keys = new Object[capacity];
counts = new int[capacity];
}
public synchronized int increase(K key, int n)
{
int id = System.identityHashCode(key) % capacity;
while (keys[id] != null && keys[id] != key) // if it's occupied, let's move to the next one!
id = (id + 1) % capacity;
if (keys[id] == null)
{
key_count++;
keys[id] = key;
if (key_count > LOAD_FACTOR * capacity)
{
resize((int) (GROW_FACTOR * capacity));
}
}
counts[id] += n;
total += n;
return counts[id];
}
public synchronized void resize(int capacity)
{
System.out.println("Resizing counters: " + this);
this.capacity = capacity;
Object[] new_keys = new Object[capacity];
int[] new_counts = new int[capacity];
for (int i = 0; i < keys.length; i++)
{
Object key = keys[i];
int count = counts[i];
int id = System.identityHashCode(key) % capacity;
while (new_keys[id] != null && new_keys[id] != key) // if it's occupied, let's move to the next one!
id = (id + 1) % capacity;
new_keys[id] = key;
new_counts[id] = count;
}
this.keys = new_keys;
this.counts = new_counts;
}
public int bump(K key)
{
return increase(key, 1);
}
public int get(K key)
{
int id = System.identityHashCode(key) % capacity;
while (keys[id] != null && keys[id] != key) // if it's occupied, let's move to the next one!
id = (id + 1) % capacity;
if (keys[id] == null)
return 0;
else
return counts[id];
}
}
Any explanations? Ideas? Suggestions?
...and, as said in the beginning, it is not for this toy example in particular but for the more general case. This same exploding behavior occurs for no reason in the more complex and larger program.

Rather than feeling helpless use a profiler! That would tell you where exactly in your code all this time is spent.

Bursting the processor cache and thrashing the Translation Lookaside Buffer (TLB) may be the problem.
For String.intern you might want to do your own single-threaded implementation.
However, I'm placing my bets on the relatively bad hash values from System.identityHashCode. It clearly isn't using the top bit, as you don't appear to get ArrayIndexOutOfBoundsExceptions. I suggest replacing that with String.hashCode.

String[] src_tokens = line_src.split("[ ,.;:?!'\"]");
Just an idea -- you are creating a new Pattern object for every line here (look at the String.split() implementation). I wonder if this is also contributing to a ton of objects that need to be garbage collected?
I would create the Pattern once, probably as a static field:
final private static Pattern TOKEN_PATTERN = Pattern.compile("[ ,.;:?!'\"]");
And then change the split line do this:
String[] src_tokens = TOKEN_PATTERN.split(line_src);
Or if you don't want to create it as a static field, as least only create it once as a local variable at the beginning of the method, before the while.

In get, when you search for a nonexistent key, search time is proportional to the size of the set of keys.
My advice: if you want a HashMap, just use a HashMap. They got it right for you.

You are filling up the Perm Gen with the string intern. Have you tried viewing the -Xloggc output?

I would guess it's just memory filling up, growing outside the processor cache, memory fragmentation and the garbage collection pauses kicking in. Have you checked memory use at all? Tried to change the heap size the JVM uses?

Try to do it in python, and run the python module from Java.
Enter all the keys in the database, and then execute the following query:
select key, count(*)
from keys
group by key
Have you tried to only iterate through the keys without doing any calculations? is it faster? if yes then go with option (2).

Can't you do this? You can get your answer in no time.

It's me, the original poster, something went wrong during registration, so I post separately. I'll try the various suggestions given.
PS for Tom Hawtin: thanks for the hints, perhaps the 'String.intern()' takes more and more time as vocabulary grows, i'll check that tomorrow, as everything else.