I've written my own math parser and for some reason it takes increasing amounts of time to parse when I tried to profile the parser.
For testing I used this input: Cmd.NUM_9,Cmd.NUM_0,Cmd.NUM_0,Cmd.DIV,Cmd.NUM_2,Cmd.ADD,Cmd.NUM_6,Cmd.MULT,Cmd.NUM_3
Single execution ~1.7ms
3000 repeats ~ 1,360ms
6000 repeats ~ 5,290ms
9000 repeats ~11,800ms
The profiler says 64% of the time was spent on this function:
this is my function to allow implicit multiplications.
private void enableImplicitMultiplication(ArrayList<Cmd> input) {
int input_size = input.size();
if (input_size<2) return;
for (int i=0; i<input_size; i++) {
Cmd cmd = input.get(i);
if (i>0) {
Cmd last = input.get(i-1);
// [EXPR1, EXPR2] => [EXPR1, MULT, EXPR2]
boolean criteria1 = Cmd.exprnCmds.contains(cmd) && Cmd.exprnCmds.contains(last);
// [CBRAC, OBRAC] => [CBRAC, MULT, OBRAC]
// [NUM_X, OBRAC] => [NUM_X, MULT, OBRAC]
boolean criteria2 = cmd==Cmd.OBRAC && (last==Cmd.CBRAC || Cmd.constantCmds.contains(last));
// [CBRAC, NUM_X] => [CBRAC, MULT, NUM_X]
boolean criteria3 = last==Cmd.CBRAC && Cmd.constantCmds.contains(cmd);
if (criteria1 || criteria2 || criteria3) {
input.add(i++, Cmd.MULT);
}
}
}
}
What's going on here??
I executed the repeats like this:
public static void main(String[] args) {
Cmd[] iArray = {
Cmd.NUM_9,Cmd.NUM_0,Cmd.NUM_0,Cmd.DIV,Cmd.NUM_2,Cmd.ADD,Cmd.NUM_6,Cmd.MULT,Cmd.NUM_3
};
ArrayList<Cmd> inputArray = new ArrayList<Cmd>(Arrays.asList(iArray));
DirtyExpressionParser parser = null;
int repeat=9000;
double starttime = System.nanoTime();
for (int i=0; i<repeat; i++) {
parser = new DirtyExpressionParser(inputArray);
}
double endtime = System.nanoTime();
System.out.printf("Duration: %.2f ms%n",(endtime-starttime)/1000000);
System.out.println(parser.getResult());
}
Constructor looks like this:
public DirtyExpressionParser(ArrayList<Cmd> inputArray) {
enableImplicitMultiplication(inputArray); //executed once for each repeat
splitOnBrackets(inputArray); //resolves inputArray into Expr objects for each bracket-group
for (Expr expr:exprArray) {
mergeAndSolve(expr);
}
}
Your microbenchmark code is altogether wrong: microbenchmarking on the JVM is a craft in its own right and is best left to dedicated tools such as jmh or Google Caliper. You don't warm up the code, don't control for GC pauses, and so on.
One detail which does come out by analyzing your code is this:
you reuse the same ArrayList for all repetitions of the function call;
each function call may insert an element to the list;
insertion is a heavyweight operation on ArrayList: the whole contents of the list after the inserted element must be copied.
You should at least create a fresh ArrayList for each invocation, but that will not make your whole methodology correct.
From our discussion in the comments I diagnose the following issue you may have with understanding your code:
In Java there is no such thing as a variable whose value is an object. The value of the variable is a reference to the object. Therefore when you say new DirtyExpressionParser(inputArray), the constructor does not receive its own private copy of the list, but rather a reference to the one and only ArrayList you have instantiated in your main method. The next constructor call gets this same list, but now modified by the earlier invocation. This is why your list is growing all the time.
Related
I'm calling a function with different values to get a particular result. Is there any way so that I can reduce calling functions like this to improve performance and reliability in Java?
func(1,6);
func(4,0);
func(2,7);
func(5,0);
func(3,0);
private static void func(int printer,int printing){
if(printing == 0){
for(int j=1;j<=7;j++){
if(j!=printer){
arrayOdd.add(j);
}
}
}else{
for(int j=1;j<=7;j++){
if(j!=printer && j!=printing)
arrayOdd.add(j);
}
}
}
My first note is that you can replace your function with
private static void func(int printer,int printing){
for(int j=1;j<=7;j++){
if(j!=printer && j!=printing)
arrayOdd.add(j);
}
}
and it will behave the same.
My second note is that calling a function repeatedly will not have any effect on performance or reliability unless you are using recursion, in which case you may get the classic stack overflow error. Why do you want/need to call the function fewer times?
If the above function can be idempotent( same inputs always produce the exact same outputs) i would suggest memoization (https://java2blog.com/memoization-example-java/).
On seeing the function outputs, it looks like you are just removing the passed values from a fixed array of 7 digits. If thats all you are looking for, just remove the element from the array and memoize the function . That way the function will execute only 64 times worst case (64 different combinations for 2 digit combinations for 0,1,2 3,4 ,5,6,7 )
you can put the printer's values and printing's values in List or array ,but you need to put the parameters manually, as long as there is no logic hold the inputs.
I mean some thing like that
int final static THE_TIMES_YOU_CALLED_THE_FUNC=5;
List<int[]> list =new ArrayList<>(THE_TIMES_YOU_CALLED_THE_FUNC);
list.add(new int []{1,6});
list.add(new int []{4,0});
list.add(new int []{2,7});
list.add(new int []{5,0});
list.add(new int []{3,0});
for(int i=0;i<list.size();i++){
funk(list.get(i)[0],list.get(i)[]);
}
}
private static void func(int printer,int printing){
if(printing == 0){
for(int j=1;j<=7;j++){
if(j!=printer){
arrayOdd.add(j);
}
}
}else{
for(int j=1;j<=7;j++){
if(j!=printer && j!=printing)
arrayOdd.add(j);
}
}
}
in the end the compiler will call the func 5 times, and I don't see an advantege of my soultion, but you want in the end to write the func one time.
I am working with interface Multiset which is then used by two different classes: ArrayListMultiset and CounterMultiset
The ArrayListMultiset simply uses the .add method to put something in the list. So in a loop like,
Multiset<String> set = new Multiset<String>();
for(int i = 0; i < 10000; i++)
{
set.add("Hello");
}
this will cause the program to add Hello to a list 10,000 times.
Next we have CounterMultiset. It stores a Pair object (another class that takes in (T, Integer), where T is the String, "Hello" and Integer is the number of times it is trying to be added. I have written it like so:
public void add(Multiset<T> item)
{
if(!contains(item))
{
Pair newpair = new Pair(item, 0);
pairs.add(newpair);
}
for(int i = 0; i < pairs.size(); i++)
{
if(pairs.get(i).getFirst() == item)
{
pairs.get(i).changeSecond();
}
}
}
changeSecond() increments the second number in the Object by 1 to show that the word Hello has tried to be added again.
My question is, is this an appropriate way to save space and time for a program? When would it be faster to use a Counter and when would it be faster to simply add "Hello" 10,000 times?
Hello is an intern string in your code.
You will not have a copy of Hello for each element of ArrayListMultiset. You will have a reference to String Pool object.
What is faster for get/put (I assume) - depends on underlying data structures.
I need to pre-populate a List with a large number of integer values.
Is there are faster way to do this other than iteration?
Current Code:
class VlanManager {
Queue<Integer> queue = Lists.newLinkedList();
public VlanManager(){
for (int i = 1; i < 4094; i++) {
queue.add(i);
}
}
This code is in the constructor of a class that is created pretty frequently so I'd like this to be as efficient (read:performance not lines of code) as possible
4094 isnt to many items to loop but if it is getting called very frequently you might look at doing something with a static variable.
private static Integer[] theList;
static {
theList = new Integer[4094];
for (int i = 1; i < 4094; i++) {
theList[i-1] = i;
}
}
then make that list a List
Queue<Integer> intQue = new LinkedList(Arrays.asList(theList));
There is a danger of using this method if you have a list of mutable objects. Heres an example of what can happen. Integers are immutable so this doesnt actually apply to your question as it stands
class MyMutableObject {
public int theValue;
}
class Test {
private static MyMutableObject[] theList;
static {
theList = new MyMutableObject[4094];
for (int i = 1; i <= 4094; i++) {
theList[i-1] = new MyMutableObject();
theList[i-1].theValue = i;
}
}
public static void main(String [] args) {
Queue<MyMutableObject> que = new LinkedList(Arrays.asList(theList));
System.out.println(que.peek().theValue); // 1
// your actually modifing the same object as the one in your static list
que.peek().theValue = -100;
Queue<MyMutableObject> que2 = new LinkedList(Arrays.asList(theList));
System.out.println(que2.peek().theValue); // -100
}
}
#Bohemian Has some good points on using a static List instead of an array, while the performance gains are very small they are none the less performance gains. Also because the 'array' is actually only ever being used as a List not an array it should be declared as such.
private static List<Integer> theList;
static {
theList = new ArrayList(4094);
for (Integer i = 0; i < 4094; i++) {
theList.add(i+1);
}
}
The fastest way would be to create a reference list (initialized using an instance block - neatly wrapping it all up in one statement):
private static final List<Integer> LIST = new ArrayList<Integer>(4094) {{
for (int i = 1; i < 4094; i++)
LIST.add(i);
}};
Then in your constructor, initialize the queue using the copy constructor:
Queue<Integer> queue;
public VlanManager(){
queue = new LinkedList<Integer>(LIST);
}
You will not write a faster implementation than what's in the JDK.
I realize this question has already been answered. But I think one important answer is missing: The fastest way to initialize a LinkedList with the values 0..4093 is .. DON'T DO IT AT ALL. Especially if speed is an issue.
What you basically are doing is creating a structure consisting of 4093 Node elements each consiting of two pointers to prev/next element and one pointer to an Integer object. Each of this Nodes must be created (and free). In addition nearly each contained Integer must be created (and freed). 'Nearly' because Java uses a cache for Integer but normally (you can change this with system properties) in the range of -127..127.
This is a lot to do in order to get a simple list of integer and if used intensively gives the GC a lot to do afterwards.
That being said there are numerous possible ways of doing this in a more efficient way. But they depend on what your concrete usage pattern is. Just to name a few:
Use an Array: boolean [] inUse' and set the taken vlan-id totrue` if it's taken
Even better use a BitSet instead of the array
Don't store which vlan is free, but which vlan is taken. I think they tend to be free and so there are much more free as there are taken ones. (this means much less to keep track of).
If you insist on using a LinkedList don't initialize it with your class but have it already initialized. This depends on how much of them you would need. You could keep a pool of them. Or perhaps your codes allows reusage of old lists. (yes, you could sort them after usage.)
Surely there are more...
All of this methods require you to build your own 'Queue' interface. But perhaps this has not to be as rich as Java's. And it really isn't that difficult. If you really use this intensively you could reach perfomance improvement factor 10x-1000x++.
A possible implementation using BitSet with an instantiation cost of nearly nothing could be:
import java.util.BitSet;
import org.testng.annotations.Test;
public class BitSetQueue {
// Represents the values 0..size-1
private final BitSet bitset;
private final int size;
private int current = 0;
private int taken = 0;
public BitSetQueue( int size ){
this.bitset = new BitSet( size );
this.size = size;
this.current = size-1;
}
public int poll(){
// prevent endless loop
if( taken == size ) return -1;
// seek for next free value.
// can be changed according to policy
while( true ){
current = (current+1)%size;
if( ! bitset.get( current ) ){
bitset.set( current );
taken++;
return current;
}
}
}
public boolean free( int num ){
if( bitset.get( num ) ){
bitset.clear( num );
taken--;
return true;
}
return false;
}
#Test
public static void usage(){
BitSetQueue q = new BitSetQueue( 4094 );
for( int i = 0; i < 4094; i++ ){
assertEquals( q.poll(), i );
}
assertEquals( q.poll(), -1 ); // No more available
assertTrue( q.free( 20 ) );
assertTrue( q.free( 51 ) );
assertEquals( q.poll(), 20 );
assertEquals( q.poll(), 51 );
}
}
What is the performance of Groovys collection methods (regarding space(!) and time) in comparison to plain Java for-loops?
Eg for this use cases:
sum() vs. for-loop with variable
each() vs. for-loop with variable
inject() vs. for-loop with variable
collect() vs. for-loop with temporary collection
findAll() vs. for-loop with temporary collection
find() vs. for-loop with variable
So, considering those results, is it advisable to use for-loops over Groovy-collection-methods in critical environments (eg. Grails-WebApp)? Are there resources regarding Groovy/Grails performance (optimization)?
Using this GBench test I got the following results for CPU-time:
user system cpu real
forLoop 2578777 67 2578844 2629592
forEachLoop 2027941 47 2027988 2054320
groovySum 3946137 91 3946228 3958705
groovyEach 4703000 0 4703000 4703000
groovyInject 4280792 108 4280900 4352287
import groovyx.gbench.BenchmarkBuilder
def testSize = 10000
def testSet = (0..testSize) as Set
def bm = new BenchmarkBuilder().run {
'forLoop' {
def n = 0
for(int i = 0; i<testSize; i++) {
n += i
}
return n
}
'forEachLoop' {
def n = 0
for(int i in testSet) {
n += i
}
return n
}
'groovySum' {
def n = testSet.sum()
return n
}
'groovyEach' {
def n = 0
testSet.each { n + it }
return n
}
'groovyInject' {
def n = testSet.inject(0) { el, sum -> sum + el }
return n
}
}
bm.prettyPrint()
Interesting benchmark. No surprise that sum() is slower. Here's how implementation looks like:
private static Object sum(Iterable self, Object initialValue, boolean first) {
Object result = initialValue;
Object[] param = new Object[1];
for (Object next : self) {
param[0] = next;
if (first) {
result = param[0];
first = false;
continue;
}
MetaClass metaClass = InvokerHelper.getMetaClass(result);
result = metaClass.invokeMethod(result, "plus", param);
}
return result;
}
As You can see it must be generic and uses meta programming. The result is bigger time cost.
The results of the benchmark You pasted are clear and pretty self descriptive. If You really need better performance it seems that better idea is to use for loops.
Disclaimer: I have looked through this
question and this question
but they both got derailed by small
details and general
optimization-is-unnecessary concerns.
I really need all the performance I
can get in my current app, which is
receiving-processing-spewing MIDI data
in realtime. Also it needs to scale up
as well as possible.
I am comparing array performance on a high number of reads for small lists to ArrayList and also to just having the variables in hand. I'm finding that an array beats ArrayList by a factor of 2.5 and even beats just having the object references.
What I would like to know is:
Is my benchmark okay? I have switched the order of the tests and number of runs with no change. I've also used milliseconds instead of nanoseconds to no avail.
Should I be specifying any Java options to minimize this difference?
If this difference is real, in this case shouldn't I prefer Test[] to ArrayList<Test> in this situation and put in the code necessary to convert them? Obviously I'm reading a lot more than writing.
JVM is Java 1.6.0_17 on OSX and it is definitely running in Hotspot mode.
public class ArraysVsLists {
static int RUNS = 100000;
public static void main(String[] args) {
long t1;
long t2;
Test test1 = new Test();
test1.thing = (int)Math.round(100*Math.random());
Test test2 = new Test();
test2.thing = (int)Math.round(100*Math.random());
t1 = System.nanoTime();
for (int i=0; i<RUNS; i++) {
test1.changeThing(i);
test2.changeThing(i);
}
t2 = System.nanoTime();
System.out.println((t2-t1) + " How long NO collection");
ArrayList<Test> list = new ArrayList<Test>(1);
list.add(test1);
list.add(test2);
// tried this too: helps a tiny tiny bit
list.trimToSize();
t1= System.nanoTime();
for (int i=0; i<RUNS; i++) {
for (Test eachTest : list) {
eachTest.changeThing(i);
}
}
t2 = System.nanoTime();
System.out.println((t2-t1) + " How long collection");
Test[] array = new Test[2];
list.toArray(array);
t1= System.nanoTime();
for (int i=0; i<RUNS; i++) {
for (Test test : array) {
test.changeThing(i);
}
}
t2 = System.nanoTime();
System.out.println((t2-t1) + " How long array ");
}
}
class Test {
int thing;
int thing2;
public void changeThing(int addThis) {
thing2 = addThis + thing;
}
}
Microbenchmarks are very, very hard to get right on a platform like Java. You definitely have to extract the code to be benchmarked into separate methods, run them a few thousand times as warmup and then measure. I've done that (code below) and the result is that direct access through references is then three times as fast as through an array, but the collection is still slower by a factor of 2.
These numbers are based on the JVM options -server -XX:+DoEscapeAnalysis. Without -server, using the collection is drastically slower (but strangely, direct and array access are quite a bit faster, indicating that there is something weird going on). -XX:+DoEscapeAnalysis yields another 30% speedup for the collection, but it's very much questionabled whether it will work as well for your actual production code.
Overall my conclusion would be: forget about microbenchmarks, they can too easily be misleading. Measure as close to production code as you can without having to rewrite your entire application.
import java.util.ArrayList;
public class ArrayTest {
static int RUNS_INNER = 1000;
static int RUNS_WARMUP = 10000;
static int RUNS_OUTER = 100000;
public static void main(String[] args) {
long t1;
long t2;
Test test1 = new Test();
test1.thing = (int)Math.round(100*Math.random());
Test test2 = new Test();
test2.thing = (int)Math.round(100*Math.random());
for(int i=0; i<RUNS_WARMUP; i++)
{
testRefs(test1, test2);
}
t1 = System.nanoTime();
for(int i=0; i<RUNS_OUTER; i++)
{
testRefs(test1, test2);
}
t2 = System.nanoTime();
System.out.println((t2-t1)/1000000.0 + " How long NO collection");
ArrayList<Test> list = new ArrayList<Test>(1);
list.add(test1);
list.add(test2);
// tried this too: helps a tiny tiny bit
list.trimToSize();
for(int i=0; i<RUNS_WARMUP; i++)
{
testColl(list);
}
t1= System.nanoTime();
for(int i=0; i<RUNS_OUTER; i++)
{
testColl(list);
}
t2 = System.nanoTime();
System.out.println((t2-t1)/1000000.0 + " How long collection");
Test[] array = new Test[2];
list.toArray(array);
for(int i=0; i<RUNS_WARMUP; i++)
{
testArr(array);
}
t1= System.nanoTime();
for(int i=0; i<RUNS_OUTER; i++)
{
testArr(array);
}
t2 = System.nanoTime();
System.out.println((t2-t1)/1000000.0 + " How long array ");
}
private static void testArr(Test[] array)
{
for (int i=0; i<RUNS_INNER; i++) {
for (Test test : array) {
test.changeThing(i);
}
}
}
private static void testColl(ArrayList<Test> list)
{
for (int i=0; i<RUNS_INNER; i++) {
for (Test eachTest : list) {
eachTest.changeThing(i);
}
}
}
private static void testRefs(Test test1, Test test2)
{
for (int i=0; i<RUNS_INNER; i++) {
test1.changeThing(i);
test2.changeThing(i);
}
}
}
class Test {
int thing;
int thing2;
public void changeThing(int addThis) {
thing2 = addThis + thing;
}
}
Your benchmark is only valid if your actual use case matches the benchmark code, i.e. very few operations on each element, so that execution time is largely determined by access time rather than the operations themselves. If that is the case then yes, you should be using arrays if performance is critical. If however your real use case involves a lot more actual computation per element, then the access time per element will become a lot less significant.
It is probably not valid. If I understand the way that JIT compilers work, compiling a method won't affect a call to that method that is already executing. Since the main method is only called once, it will end up being interpreted, and since most of the work is done in the body of that method, the numbers you get won't be particularly indicative of normal execution.
JIT compilation effects may go some way to explain why the no collections case was slower that the arrays case. That result is counter-intuitive, and it places a doubt on the other benchmark result that you reported.