We Keep Coding

Large Optimized IO processing in Java

An input n of the order 10^18 and output should be the sum of all the numbers whose set bits is only 2. For e.g. n = 5 setbit is 101--> 2 set bits. For n = 1234567865432784,How can I optimize the below code?
class TestClass
{
public static void main(String args[])
{
long N,s=0L;
Scanner sc = new Scanner(System.in);
N=sc.nextLong();
for(long j = 1; j<=N; j++)
{
long b = j;
int count = 0;
while(b!=0)
{
b = b & (b-1);
count++;
}
if(count == 2)
{
s+=j;
count = 0;
}
else
{
count = 0;
continue;
}
}
System.out.println(s%1000000007);
s=0L;
}
}

Java has a function
if (Integer.bitCount(i) == 2) { ...
However consider a bit: that are a lot of numbers to inspect.
What about generating all numbers that have just two bits set?
Setting the ith and jth bit of n:
int n = (1 << i) | (1 << j); // i != j
Now consider 31² steps, not yet 1000 with N steps.
As this is homework my advise:
Try to turn the problem around, do the least work, take a step back, find the intelligent approach, search the math core. And enjoy.
Next time, do not spoil yourself of success moments.

As you probably had enough time to think about Joop Eggen's suggestion,
here is how i would do it (which is what Joop described i think):
import java.util.Scanner;
public class Program {
public static void main(String[] args) {
Scanner sc = new Scanner(System.in);
long n = sc.nextLong();
long sum = 0;
for (int firstBitIndex = 0; firstBitIndex < 64; firstBitIndex++) {
long firstBit = 1L << firstBitIndex;
if (firstBit >= n)
break;
for (int secondBitIndex = firstBitIndex + 1; secondBitIndex < 64; secondBitIndex++) {
long value = firstBit | (1L << secondBitIndex);
if (value > n)
break;
sum += value;
}
}
System.out.println(sum % 1000000007);
sc.close();
}
}

Java provides the class BigInteger, which includes a method nextProbablePrime(). This means you could do something like this:
BigInteger n = new BigInteger(stringInputN);
BigInteger test = BigInteger.valueOf(2);
BigInteger total = BigInteger.valueOf(0);
while (test.compareTo(n) < 0){
total = total.add(test);
test = test.nextProbablePrime();
}
System.out.println(total);
This this has an extremely low probability of getting the wrong answer (but nonzero), so you might want to run it twice just to doublecheck. It should be faster than manually iterating it by hand though.

Memoization of a Recursive Search

I am trying to solve a problem in which you have to count the number of possible bar codes you can make given specific parameters. I solved the problem recursively and am able to get the correct answer every time. However, my program is dreadfully slow. I tried to rectify this using a technique I read about called memoization but my program still crawls when given certain input (ex: 10, 10, 10). Here's the code in java.
Does anybody have any idea what I'm doing wrong here?
import java.util.Scanner;
//f(n, k, m) = sum (1 .. m) f(n - i, k - 1, m)
public class BarCode { public static int[][] memo;
public static int count(int units, int bars, int width) {
int sum = 0;
if (units >= 0 && memo[units][bars] != -1) //if the value has already been calculated return that value
return memo[units][bars];
for (int i = 1; i <= width; ++i) {
if (units == 0 && bars == 0)
return 1;
else if (bars == 0)
return 0;
else {
sum += count(units - i, bars - 1, width);
}
}
if (units > -1)
memo[units][bars] = sum;
return sum;
}
public static void main(String[] args) {
Scanner in = new Scanner(System.in);
//while (in.hasNext()) {
int num = in.nextInt();
int bars = in.nextInt();
int width = in.nextInt();
memo = new int[51][51];
for (int i = 0; i < memo.length; ++i) {
for (int j = 0; j < memo.length; ++j)
memo[i][j] = -1;
}
int sum = 0;
sum += count(num, bars, width);
System.out.println(sum);
//}
in.close();
}
}
TL:DR My memoization of a recursive search is too slow. Help!

You exclude all results from count calls with units < 0 from memoization:
if (units > -1)
memo[units][bars] = sum;
This leads to a lot of unnecessary calls to count for these values.
To include all cases, you could use a HashMap with a key generated from units and bars values. I used a string generated from units and bars like this:
//f(n, k, m) = sum (1 .. m) f(n - i, k - 1, m)
public class BarCode {
public static Map<String, Integer> memo = new HashMap<>();
public static int count(int units, int bars, int width) {
int sum = 0;
final String key = units + " " + bars;
Integer memoSum = memo.get(key);
if (memoSum != null) {
return memoSum.intValue();
}
for (int i = 1; i <= width; ++i) {
if (units == 0 && bars == 0)
return 1;
else if (bars == 0)
return 0;
else {
sum += count(units - i, bars - 1, width);
}
}
memo.put(key, Integer.valueOf(sum));
return sum;
}
public static void main(String[] args) {
Scanner in = new Scanner(System.in);
int num = in.nextInt();
int bars = in.nextInt();
int width = in.nextInt();
memo = new HashMap<>();
int sum = 0;
sum += count(num, bars, width);
System.out.println(sum);
in.close();
}
}
For example, this brings the number of calls to count down from over 6 million to 4,150 for the input values "10 10 10" with 415 entries saved in the Map.

Your memoization implementation looks to be valid. It might help some, but the real problem here is your choice of algorithm.
From my cursory inspection of your code, on average a call to your count method will loop through width number of times. and each time it loops through, it goes a layer deeper by calling count again. It also looks like it's going to loop down bars layers deeper from the first layer. If my asymptotic analysis a few fingers of scotch in is correct, this would result in an algorithm which has a O(width^bars) runtime complexity. As you increase your input parameters, especially bars, the amount of steps your application needs to take in order to calculate your answer will increase greatly (exponentially, in the case of bars).
Your memoization will reduce the number of duplicate calculations needed, but each value being memoized will still need to be calculated at least once for the memoization to help. So with or without the memoization, you're still dealing with a non-polynomial time complexity, and that always spells bad performance.
You might want to consider looking for a more efficient approach. Instead of trying to count the number of bar code combinations, perhaps try using combinatorics to try to calculate it. For example, I could try to figure out the number of lowercase character strings (using only chars a-z) I can make for a string of length n by generating all of them and counting how many of them there are, but that will have an exponential time complexity and will not be performant. On the other hand, I know basic combinatorics tells me that the formula for the number of strings I can create is 26^n (26 choices in each position, and n positions), which the computer can easily evaluate quickly.
Look for a similar approach for computing the number of bar codes.

What should be the optimal way of solving Recurrence relation for really Huge number greater than Integer maximum value

I want to find the Nth number of the Recurrence Equation
T(n)=T(n-1)+3T(n-2)+3T(n-3)+(n-4),T(1)=T(4)=1,T(2)=T(3)=3
so if suppose you entered 2,5,9 as input, output should be T(2)=3,T(5)=20,T(9)=695
what I did is create an array of size equal to maximum of all input value and storing solution of T(i) at index i.Then look up into the array for specific index. eg array[3] for T(3),array[5] for T(5),etc
The code worked fine till maximum number is not greater than maximum integer value system can hold i.e
Integer.MAXValue.
Because the index of array can only be integer then
if number is n=1855656959555656 what should be the best way to find the solution of
T(1855656959555656)?
as clearly I cant create an array of size=1855656959555656..
I have even tried BigInteger from java.Math but with no success.
I have to find some other approach.please suggest some ideas..
Thanks

you do not need to store every T(i), you only need to store 3 values T(i-1), T(i-2), T(i-3). While looping over i, check if the current i should be part of your output, if so put it out immediately or save it to an "output"-array.
edit: this part is quite inefficient. You check in every iteation EVERY needed output.
for (int k = 0; k < arr.length; ++k) {
if (count == arr[k])
T[k] = temp[i];
else if (arr[k] == 1)
T[k] = 1;
else if (arr[k] == 2)
T[k] = 3;
else if (arr[k] == 3)
T[k] = 3;
else if (arr[k] == 4)
T[k] = 1;
}
so your code runs in time (max*arr.length) you can reduce it to only (max). Use a HashMap with key=neededPosition (=count) value=position in arr
Init the map like this:
Map<Long, Integer> map = new HashMap<Long, Integer>();
for (int i = 0; i < arr.length; i++) {
map.put(arr[i], i);
}
if (map.containsKey(count)) {
T[map.get(count)] = temp[i]
}
check the values 1-4 just once after the whole thing!

Not possible. The array size can be a maximum of Integer.MAX_VALUE (minus something usually 5 or 8, depending on the JVM capabilities). Why?. The index for an Array should be an integer thats a limitation.

It can't be done. So you need to solve the problem by introducing a sharding mechanism. The simplest way would be to just have arrays of arrays with a fixed length.

Edit: You really do not need this much storage for your problem at hand (as pointed out in another answer; this code fragment avoids arrays altogether to avoid bounds checks / indirection):
public void t(long n) {
if (n < 5) {
return (n == 2 || n == 3) ? 3 : 1;
}
long i = 5; // Initialize variables for n == 5;
long tn_1 = 1; // T(n-1) = T(4) = 1;
long tn_2 = 3; // T(n-2) = T(3) = 3;
long tn_3 = 1; // T(n-3) = T(2) = 1;
long tn_4 = 3; // T(n-4) = T(1) = 3;
while (true) {
long tn = tn_1 + 3*tn_2 + 3*tn_3 + tn_4;
if (i++ == n) {
return tn;
}
tn_4 = tn_3;
tn_3 = tn_2;
tn_2 = tn_1;
tn_1 = tn;
}
}
To answer the question in the title anyway:
If your array is sparse, use a map (TreeMap or HashMap) of Long or BigInteger:
Map<Long,Long> t = new TreeMap<Long,Long>()
The memory consumption of sparse arrays depends on the number of elements actually stored, so you may want to delete values from the map that are no longer needed.
If your array is not sparse, use a 2-level array (memory consumption will depend on the pre-allocated size only):
public class LongArray {
static final long BLOCK_SIZE = 0x40000000;
long[][] storage;
public LongArray(long size) {
long blockCount = (size + BLOCK_SIZE - 1) / BLOCK_SIZE;
storage = new long[][(int) blockCount];
for (long i = 0; i < blockCount; i++) {
if (i == blockCount - 1) {
storage[i] = new long[(int) size - BLOCK_SIZE * (blockCount - 1)];
} else {
storage[i] = new long[(int) BLOCK_SIZE];
}
}
}
public long get(long index) {
return storage[(int) (index / BLOCK_SIZE)][(int) (index % BLOCK_SIZE)];
}
public void put(long index, long value) {
storage[(int) (index / BLOCK_SIZE)][(int) (index % BLOCK_SIZE)] = value;
}
}
In both cases, use t.get(index) and t.put(index, value) instead of t[index] to access your array (if t is the name of the array).

You can do one thing. Check if the value of n is equal to 1855656959555656 in the beginning or if its multiple. Suppose, the value of n is twice of 1855656959555656. Then you can create two arrays and link them together virtually. This should solve your problem but it will involve a lot of overhead.

Use recursive call:
int T(int n){
if (n==1 || n==4){
return 1;
} else if (n==2 || n==3){
return 3;
} else {
return T(n-1)+3*T(n-2)+3T*(n-3)+T(n-4);
}
}
Edit: Time consumming. Won't work with large numbers

Java: how ot optimize sum of big array

I try to solve one problem on codeforces. And I get Time limit exceeded judjment. The only time consuming operation is calculation sum of big array. So I've tried to optimize it, but with no result.
What I want: Optimize the next function:
//array could be Integer.MAX_VALUE length
private long canocicalSum(int[] array) {
int sum = 0;
for (int i = 0; i < array.length; i++)
sum += array[i];
return sum;
}
Question1 [main]: Is it possible to optimize canonicalSum?
I've tried: to avoid operations with very big numbers. So i decided to use auxiliary data. For instance, I convert array1[100] to array2[10], where array2[i] = array1[i] + array1[i+1] + array1[i+9].
private long optimizedSum(int[] array, int step) {
do {
array = sumItr(array, step);
} while (array.length != 1);
return array[0];
}
private int[] sumItr(int[] array, int step) {
int length = array.length / step + 1;
boolean needCompensation = (array.length % step == 0) ? false : true;
int aux[] = new int[length];
for (int i = 0, auxSum = 0, auxPointer = 0; i < array.length; i++) {
auxSum += array[i];
if ((i + 1) % step == 0) {
aux[auxPointer++] = auxSum;
auxSum = 0;
}
if (i == array.length - 1 && needCompensation) {
aux[auxPointer++] = auxSum;
}
}
return aux;
}
Problem: But it appears that canonicalSum is ten times faster than optimizedSum. Here my test:
#Test
public void sum_comparison() {
final int ARRAY_SIZE = 100000000;
final int STEP = 1000;
int[] array = genRandomArray(ARRAY_SIZE);
System.out.println("Start canonical Sum");
long beg1 = System.nanoTime();
long sum1 = canocicalSum(array);
long end1 = System.nanoTime();
long time1 = end1 - beg1;
System.out.println("canon:" + TimeUnit.MILLISECONDS.convert(time1, TimeUnit.NANOSECONDS) + "milliseconds");
System.out.println("Start optimizedSum");
long beg2 = System.nanoTime();
long sum2 = optimizedSum(array, STEP);
long end2 = System.nanoTime();
long time2 = end2 - beg2;
System.out.println("custom:" + TimeUnit.MILLISECONDS.convert(time2, TimeUnit.NANOSECONDS) + "milliseconds");
assertEquals(sum1, sum2);
assertTrue(time2 <= time1);
}
private int[] genRandomArray(int size) {
int[] array = new int[size];
Random random = new Random();
for (int i = 0; i < array.length; i++) {
array[i] = random.nextInt();
}
return array;
}
Question2: Why optimizedSum works slower than canonicalSum?

As of Java 9, vectorisation of this operation has been implemented but disabled, based on benchmarks measuring the all-in cost of the code plus its compilation. Depending on your processor, this leads to the relatively entertaining result that if you introduce artificial complications into your reduction loop, you can trigger autovectorisation and get a quicker result! So the fastest code, for now, assuming numbers small enough not to overflow, is:
public int sum(int[] data) {
int value = 0;
for (int i = 0; i < data.length; ++i) {
value += 2 * data[i];
}
return value / 2;
}
This isn't intended as a recommendation! This is more to illustrate that the speed of your code in Java is dependent on the JIT, its trade-offs, and its bugs/features in any given release. Writing cute code to optimise problems like this is at best vain and will put a shelf life on the code you write. For instance, had you manually unrolled a loop to optimise for an older version of Java, your code would be much slower in Java 8 or 9 because this decision would completely disable autovectorisation. You'd better really need that performance to do it.

Question1 [main]: Is it possible to optimize canonicalSum?
Yes, it is. But I have no idea with what factor.
Some things you can do are:
use the parallel pipelines introduced in Java 8. The processor has instruction for doing parallel sum of 2 arrays (and more). This can be observed in Octave when you sum two vectors with ".+" (parallel addition) or "+" it is way faster than using a loop.
use multithreading. You could use a divide and conquer algorithm. Maybe like this:
divide the array into 2 or more
keep dividing recursively until you get an array with manageable size for a thread.
start computing the sum for the sub arrays (divided arrays) with separate threads.
finally add the sum generated (from all the threads) for all sub arrays together to produce final result
maybe unrolling the loop would help a bit, too. By loop unrolling I mean reducing the steps the loop will have to make by doing more operations in the loop manually.
An example from http://en.wikipedia.org/wiki/Loop_unwinding :
for (int x = 0; x < 100; x++)
{
delete(x);
}
becomes
for (int x = 0; x < 100; x+=5)
{
delete(x);
delete(x+1);
delete(x+2);
delete(x+3);
delete(x+4);
}
but as mentioned this must be done with caution and profiling since the JIT could do this kind of optimizations itself probably.
A implementation for mathematical operations for the multithreaded approach can be seen here.
The example implementation with the Fork/Join framework introduced in java 7 that basically does what the divide and conquer algorithm above does would be:
public class ForkJoinCalculator extends RecursiveTask<Double> {
public static final long THRESHOLD = 1_000_000;
private final SequentialCalculator sequentialCalculator;
private final double[] numbers;
private final int start;
private final int end;
public ForkJoinCalculator(double[] numbers, SequentialCalculator sequentialCalculator) {
this(numbers, 0, numbers.length, sequentialCalculator);
}
private ForkJoinCalculator(double[] numbers, int start, int end, SequentialCalculator sequentialCalculator) {
this.numbers = numbers;
this.start = start;
this.end = end;
this.sequentialCalculator = sequentialCalculator;
}
#Override
protected Double compute() {
int length = end - start;
if (length <= THRESHOLD) {
return sequentialCalculator.computeSequentially(numbers, start, end);
}
ForkJoinCalculator leftTask = new ForkJoinCalculator(numbers, start, start + length/2, sequentialCalculator);
leftTask.fork();
ForkJoinCalculator rightTask = new ForkJoinCalculator(numbers, start + length/2, end, sequentialCalculator);
Double rightResult = rightTask.compute();
Double leftResult = leftTask.join();
return leftResult + rightResult;
}
}
Here we develop a RecursiveTask splitting an array of doubles until
the length of a subarray doesn't go below a given threshold. At this
point the subarray is processed sequentially applying on it the
operation defined by the following interface
The interface used is this:
public interface SequentialCalculator {
double computeSequentially(double[] numbers, int start, int end);
}
And the usage example:
public static double varianceForkJoin(double[] population){
final ForkJoinPool forkJoinPool = new ForkJoinPool();
double total = forkJoinPool.invoke(new ForkJoinCalculator(population, new SequentialCalculator() {
#Override
public double computeSequentially(double[] numbers, int start, int end) {
double total = 0;
for (int i = start; i < end; i++) {
total += numbers[i];
}
return total;
}
}));
final double average = total / population.length;
double variance = forkJoinPool.invoke(new ForkJoinCalculator(population, new SequentialCalculator() {
#Override
public double computeSequentially(double[] numbers, int start, int end) {
double variance = 0;
for (int i = start; i < end; i++) {
variance += (numbers[i] - average) * (numbers[i] - average);
}
return variance;
}
}));
return variance / population.length;
}

If you want to add N numbers then the runtime is O(N). So in this aspect your canonicalSum can not be "optimized".
What you can do to reduce runtime is make the summation parallel. I.e. break the array to parts and pass it to separate threads and in the end sum the result returned by each thread.
Update: This implies multicore system but there is a java api to get the number of cores

java - how to reduce execution time for this program [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 8 years ago.
Improve this question
int n, k;
int count = 0, diff;
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
String[] input;
input = br.readLine().split(" ");
n = Integer.parseInt(input[0]);
int[] a = new int[n];
k = Integer.parseInt(input[1]);
input = br.readLine().split(" ");
for (int i = 0; i < n; i++) {
a[i] = Integer.parseInt(input[i]);
for (int j = 0; j < i; j++) {
diff = a[j] - a[i];
if (diff == k || -diff == k) {
count++;
}
}
}
System.out.print(count);
This is a sample program where I am printing particular difference count, where n range is <=100000
Now problem is to decrease execution for this program. How can I make it better to reduce running time.
Thanks in advance for suggestions

Read the numbers from a file and put them in a Map (numbers as keys, their frequencies as values). Iterate over them once, and for each number check if the map contains that number with k added. If so, increase your counter. If you use a HashMap it's O(n) that way, instead of your algorithm's O(n^2).
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
int k = Integer.parseInt(br.readLine().split(" ")[1]);
Map<Integer, Integer> readNumbers = new HashMap<Integer, Integer>();
for (String aNumber : br.readLine().split(" ")) {
Integer num = Integer.parseInt(aNumber);
Integer freq = readNumbers.get(num);
readNumbers.put(num, freq == null ? 1 : freq + 1);
}
int count = 0;
for (Integer aNumber : readNumbers.keySet()) {
int freq = readNumbers.get(aNumber);
if (k == 0) {
count += freq * (freq - 1) / 2;
} else if (readNumbers.containsKey(aNumber + k)) {
count += freq * readNumbers.get(aNumber + k);
}
}
System.out.print(count);
EDIT fixed for duplicates and k = 0

Here is a comparison of #Socha23's solution using HashSet, TIntIntHashSet and the original solution.
For 100,000 numbers I got the following (without the reading and parsing)
For 100 unique values, k=10
Set: 89,699,743 took 0.036 ms
Trove Set: 89,699,743 took 0.017 ms
Loops: 89,699,743 took 3623.2 ms
For 1000 unique values, k=10
Set: 9,896,049 took 0.187 ms
Trove Set: 9,896,049 took 0.193 ms
Loops: 9,896,049 took 2855.7 ms
The code
import gnu.trove.TIntIntHashMap;
import gnu.trove.TIntIntProcedure;
import java.util.HashMap;
import java.util.Map;
import java.util.Random;
class Main {
public static void main(String... args) throws Exception {
Random random = new Random(1);
int[] a = new int[100 * 1000];
int k = 10;
for (int i = 0; i < a.length; i++)
a[i] = random.nextInt(100);
for (int i = 0; i < 5; i++) {
testSet(a, k);
testTroveSet(a, k);
testLoops(a, k);
}
}
private static void testSet(int[] a, int k) {
Map<Integer, Integer> readNumbers = new HashMap<Integer, Integer>();
for (int num : a) {
Integer freq = readNumbers.get(num);
readNumbers.put(num, freq == null ? 1 : freq + 1);
}
long start = System.nanoTime();
int count = 0;
for (Integer aNumber : readNumbers.keySet()) {
if (readNumbers.containsKey(aNumber + k)) {
count += (readNumbers.get(aNumber) * readNumbers.get(aNumber + k));
}
}
long time = System.nanoTime() - start;
System.out.printf("Set: %,d took %.3f ms%n", count, time / 1e6);
}
private static void testTroveSet(int[] a, final int k) {
final TIntIntHashMap readNumbers = new TIntIntHashMap();
for (int num : a)
readNumbers.adjustOrPutValue(num, 1,1);
long start = System.nanoTime();
final int[] count = { 0 };
readNumbers.forEachEntry(new TIntIntProcedure() {
#Override
public boolean execute(int key, int keyCount) {
count[0] += readNumbers.get(key + k) * keyCount;
return true;
}
});
long time = System.nanoTime() - start;
System.out.printf("Trove Set: %,d took %.3f ms%n", count[0], time / 1e6);
}
private static void testLoops(int[] a, int k) {
long start = System.nanoTime();
int count = 0;
for (int i = 0; i < a.length; i++) {
for (int j = 0; j < i; j++) {
int diff = a[j] - a[i];
if (diff == k || -diff == k) {
count++;
}
}
}
long time = System.nanoTime() - start;
System.out.printf("Loops: %,d took %.1f ms%n", count, time / 1e6);
}
private static long free() {
return Runtime.getRuntime().freeMemory();
}
}

Since split() uses regular expressions to split a string, you should meassure whether StringTokenizer would speed up things.

You are trying to find elements which have difference k. Try this:
Sort the array.
You can do it in one pass after sorting by having two pointers and adjusting one of them depending on if the difference is bigger or smaller than k

A sparse map for the values, with their frequency of occurrence.
SortedMap<Integer, Integer> a = new TreeMap<Integer, Integer>();
for (int i = 0; i < n; ++i) {
int value = input[i];
Integer old = a.put(value, 1);
if (old != null) {
a.put(value, old.intValue() + 1);
}
}
for (Map.Entry<Integer, Integer> entry : a.entrySet()) {
Integer freq = a.get(entry.getKey() + k);
count += entry.getValue() * freq; // N values x M values further on.
}
This O(n).
Should this be too costly, you could sort the input array and do something similar.

I don't understand why you have one loop inside another. It's O(n^2) that way.
You also mingle reading in this array of ints with getting this count. I'd separate the two - read the whole thing in and then sweep through and get the difference count.
Perhaps I'm misunderstanding what you're doing, but it feels like you're re-doing a lot of wok in that inside loop.

Why not use java.util.Scanner clas instead of BufferReader.
for example :-
Scanner sc = new Scanner(System.in);
int number = sc.nextInt();
this may work faster as their are less wrappers involved.... See this link

Use sets and maps, as other users have already explained, so I won't reiterate their suggestions again.
I will suggest something else.
Stop using String.split. It compiles and uses a regular expression.
String.split has this line in it: Pattern.compile(expr).split(this).
If you want to split along a single character, you could write your own function and it would be much faster. I believe Guava (ex-Google collections API) has String split function which splits on characters without using a regular expression.