We Keep Coding

Optimization: Finding the best Simple Moving Average takes too much time

I've created a simple Spring-Application with a MySQL-DB.
In the DB there are 20 years of stock data (5694 lines):
The Goal is to find the Best Moving Average (N) for that 20 years of stock data. Inputs are the closing prices of every trading day.
The calculated average depends on the N. So p.e. if N=3 the average of a reference day t, is given by ((t-1)+(t-2)+(t-3)/N).
Output is the Best Moving Average (N) and the Result you made with all the buying & selling transactions of the Best N.
I did not find a proper algorithm in the Internet, so I implemented the following:
For every N (249-times) the program does the following steps:
SQL-Query: calculates averages & return list
#Repository
public interface StockRepository extends CrudRepository<Stock, Integer> {
/*
* This sql query calculate the moving average of the value n
*/
#Query(value = "SELECT a.date, a.close, Round( ( SELECT SUM(b.close) / COUNT(b.close) FROM stock AS b WHERE DATEDIFF(a.date, b.date) BETWEEN 0 AND ?1 ), 2 ) AS 'avg' FROM stock AS a ORDER BY a.date", nativeQuery = true)
List<AverageDTO> calculateAverage(int n);
Simulate buyings & sellings – > calculate result
Compare result with bestResult
Next N
#RestController
public class ApiController {
#Autowired
private StockRepository stockRepository;
#CrossOrigin(origins = "*")
#GetMapping("/getBestValue")
/*
* This function tries all possible values in the interval [min,max], calculate
* the moving avg and simulate the gains for each value to choose the best one
*/
public ResultDTO getBestValue(#PathParam("min") int min, #PathParam("max") int max) {
Double best = 0.0;
int value = 0;
for (int i = min; i <= max; i++) {
Double result = simulate(stockRepository.calculateAverage(i));
if (result > best) {
value = i;
best = result;
}
}
return new ResultDTO(value, best);
}
/*
* This function get as input the close and moving average of a stock and
* simulate the buying/selling process
*/
public Double simulate(List<AverageDTO> list) {
Double result = 0.0;
Double lastPrice = list.get(0).getClose();
for (int i = 1; i < list.size(); i++) {
if (list.get(i - 1).getClose() < list.get(i - 1).getAvg()
&& list.get(i).getClose() > list.get(i).getAvg()) {
// buy
lastPrice = list.get(i).getClose();
} else if (list.get(i - 1).getClose() > list.get(i - 1).getAvg()
&& list.get(i).getClose() < list.get(i).getAvg()) {
// sell
result += (list.get(i).getClose() - lastPrice);
lastPrice = list.get(i).getClose();
}
}
return result;
}
}
When I put Min=2 and Max=250 it takes 45 minutes to finish.
Since, I'm a beginner in Java & Spring I do not know how I can optimize it.
I'm happy for every input.

This problem is equivalent with finding the best moving N sum. Simply then divide by N. Having such a slice, then the next slice subtracts the first value and adds a new value to the end. This could lead to an algorithm for finding local growths with a[i + N] - a[i] >= 0.
However in this case a simple sequential ordered query with
double[] slice = new double[N];
double sum = 0.0;
suffices. (A skipping algorithm on a database is probably too complicated.)
Simply walk through the table keeping the slice as window, keeping N values and keys, and maintaining the maximum upto now.
Use the primitive type double instead of the object wrapper Double.
If the database transport is a serious factor, a stored procedure would do. Keeping a massive table as really many entities just for a running maximum is unfortunate.
It would be better to have a condensed table or better field with the sum of N values.

Java adding objects according to ratios and population sizes

My project is a simple agent based simulation that models the spread of disease. The simulation consists of agents and map, the classes that belong to agents are: Civil, Medic and Animal. Currently I'm trying to fill the map according to the size of populations: noMedics, noCivils, noAnimals and amount of healthy, ill, and immune agents that should add up to total population of both humans and animals.
The way that health status of an agent is determined is simple - random.nextInt(3), 0 being healthy, 1 ill and 2 immune. The way that I would usually fill the map with one of the agent class is as follows:
for (int i = 0; i < noAnimals; i++) {
IAgent animal = new Animal(map, rnd.nextInt(3), 2, 3);
agentList.add(animal);
}
However the issue arises whenever I would try to implement the remaining parameters of simulation that is: noHealthy, noImmune, noIll. I can't seem to find a loop or condition that would fullfill my need which is to "fairly" fill the map with all agents of in the given createAgents() method. Create agents takes 6 parameters: noHealthy, noImmune, noIll, noAnimals, noCivil, noMedics.
I've tried few things already but the one that compiles and runs correctly so far is as follows: I'm creating a loop that runs from currNoImmune until noImmune from parameters and whenever a condition sumForImmune < halfNoAnimals && currNoImmune <= noImmune is fulfilled it adds an animal to the simulation and increments sumForImmune and currNoImmune once. The reverse check is done for civlians sumForImmune >= halfNoAnimals && currNoImmune <= noImmune incrementing the same variables and adding a civil to the simulation. Here is the code for method I've described:
while (sumForImmune <= noImmune) {
if (sumForImmune < halfNoAnimals && currNoImmune <= noImmune) {
agentList.add(new Animal(map, 2, 0, 2));
sumForImmune++;
currNoImmune++;
}
if (sumForImmune >= halfNoAnimals && currNoImmune <= noImmune) {
agentList.add(new Civil(map, 2, 0, 2));
sumForImmune++;
currNoImmune++;
}
}
Then there are two loops that run until noIll and noHealthy and that's how agents are created so far. It works however not quite how I hoped it would. Numbers that are passed as arugments to createAgents() aren't being reflected on the map for all possible inputs. I realize that this task is beyond my capabilities since I've spend a good amount of time trying to figure it out, despite that I would still love to understand how it's done and how it can be achieved.
What I mean by fairly is as close to 50:50 as possible - whenever user inputs an uneven number of 3 immune, 1 animal and 2 civil agents there should be 2 immune civilians and 1 immune animal. Same logic shoud be extended to the missing parameters that is healthy and ill agents.
Edit:
What I mean by that mess written above is that I need an algorithm to place agents according to the ratios determined by noHealthy:noIll:noImmune for both population of Civilians (noCivils) and population of Animals (noAnimals). Taking into account that Medics are already immune so noImmune should shrink by the number of Medics present in the simulation.
Edit2:
I've played around with the maths a bit and this is what I managed to get but there is still issue with 1:1:1 ratios as they don't give expected results for given population sizes. One more thing is this doesn't account for medics yet, just so it doesn't mess the ratios and makes the logic a bit easier.
void createAgents(int noAnimals, int noCivil, noIll, noImmune, noHealthy) {
double sumTheRatio = noHealthy + noIll + noImmune;
double animalsPart = noAnimals / sumTheRatio;
double numHealthyAnimals = noHealthy * animalsPart;
double numIllAnimals = noIll * animalsPart;
double numImmuneAnimals = noImmune * animalsPart;
double civilPart = noCivil / sumTheRatio;
double numHealthyCivil = noHealthy * civilPart;
double numIllCivil = noIll * civilPart;
double numImmuneCivil = noImmune * civilPart;
//The only issue is when ratios of agent health states are 1:1:1
//- for example for ratios like 18:18:18 and 26 civilians 28 animals will
// give 27 agents for both animals and civilians (+/- 1 from expected numbers)
//- another example 7:7:7 and 1 civil and 20 animals will give 0 agents
// for civilians and 21 agents for animals (+/- 1 from expected numbers)
//- another example 14:14:14 and 38 civilians and 4 animals will give 39
// agents for civilians and 3 agents for animals (+/- 1 from expected numbers)
System.out.println("Rounded ratio number for animals:"
+ "\n healthy - " + Math.round(numHealthyAnimals)
+ " ill - " + Math.round(numIllAnimals)
+ " immune - " + Math.round(numImmuneAnimals));
System.out.println("Rounded ratio number for civilians:"
+ "\n healthy - " + Math.round(numHealthyCivil)
+ " ill - " + Math.round(numIllCivil)
+ " immune - " + Math.round(numImmuneCivil));
}
Then simply iterating to: Math.round(numHealthyCivil), Math.round(numIllCivil), Math.round(numImmuneCivil) and adding respective agent with each iteration.
Is there a way to adjust this algorithm or perhaps there is a need for different function responsible for agents creation whenever ratios are 1:1:1?

You say that Medics should have an Immune health status. The remaining problem is therefore to assign Civilians and Animals a health status of Healthy, Ill, or Immune such that:
noCivils + noAnimals + noMedics = noHealthy + noIll + noImmune
One way of doing this would be to create a health status array noCivils + noAnimals long and populate it with noHealthy, noIll, and noImmune-noMedics elements of the corresponding type (0, 1, 2). You then randomly shuffle this array and use the values to assign a health status to Civilians and Animals in turn.
Here's some Java code to illustrate:
static void createAgents(int noHealthy, int noImmune, int noIll,
int noAnimals, int noCivil, int noMedics)
{
for (int i=0; i<noMedics; i++) {
// create and add Medic, with health status Immune
}
int remImmune = noImmune - noMedics;
assert noCivil + noAnimals == noHealthy + noIll + remImmune;
int caAgents = noCivil + noAnimals;
Integer[] agentStatus = new Integer[caAgents];
Arrays.fill(agentStatus, 0, noHealthy, 0);
Arrays.fill(agentStatus, noHealthy, noHealthy+noIll, 1);
Arrays.fill(agentStatus, noHealthy+noIll, noHealthy+noIll+remImmune, 2);
Collections.shuffle(Arrays.asList(agentStatus));
int j = 0;
for (int i=0; i<noAnimals; i++, j++) {
// create and add Animal, with health status status[j]
}
for (int i=0; i<noCivil; i++, j++) {
// create and add Civilian, with health status status[j]
}
}

Passing arrival rate and think time to a closed-circuit queue in PDQ (Pretty Damn Quick)

I am trying to simulate a road segment using PDQ, by providing the number of vehicles and arrival rate at the intersection node among other parameters.
In order to achieve this, I have opted for a closed-circuit M/M/1 queue due to the limit imposed by the vehicle count.
I have consulted the PDQ manual and have found no way to pass the arrival rate to closed-circuit queues. The manual also prescribes a user think time, which I could not connect to traffic flow theory.
Here is what I have so far -
def simulate(pdq: PDQ) = {
var l: Int = 0
for (l <- 0 until sLanes) {
val node = toString + "l" + l
val workload = node + "w"
pdq.CreateClosed(workload, Job.TERM, vehicles / l, )
pdq.CreateNode(node, defs.CEN, defs.FCFS)
pdq.SetDemand(node, workload, serviceTime)
pdq.SetTUnit("Minutes")
pdq.SetWUnit("Vehicles")
}
}

Java and Increasing the Efficiency of Genetic Algorithms

I was wondering if I could get some advice on increasing the overall efficiency of a program that implements a genetic algorithm. Yes this is an assignment question, but I have already completed the assignment on my own and am simply looking for a way to get it to perform better
Problem Description
My program at the moment reads a given chain made of the types of constituents, h or p. (For example: hphpphhphpphphhpphph) For each H and P it generated a random move (Up, Down, Left, Right) and adds the move to an arrayList contained in the "Chromosome" Object. At the start the program is generating 19 moves for 10,000 Chromosomes
SecureRandom sec = new SecureRandom();
byte[] sbuf = sec.generateSeed(8);
ByteBuffer bb = ByteBuffer.wrap(sbuf);
Random numberGen = new Random(bb.getLong());
int numberMoves = chromosoneData.length();
moveList = new ArrayList(numberMoves);
for (int a = 0; a < numberMoves; a++) {
int randomMove = numberGen.nextInt(4);
char typeChro = chromosoneData.charAt(a);
if (randomMove == 0) {
moveList.add(Move.Down);
} else if (randomMove == 1) {
moveList.add(Move.Up);
} else if (randomMove == 2) {
moveList.add(Move.Left);
} else if (randomMove == 3) {
moveList.add(Move.Right);
}
}
After this comes the selection of chromosomes from the Population to crossover. My crossover function selections the first chromosome at random from the fittest 20% of the population and the other at random from outside of the top 20%. The chosen chromosomes are then crossed and a mutation function is called. I believe the area in which I am taking the biggest hit is calculating the fitness of each Chromosome. Currently my fitness function creates a 2d Array to act as a grid, places the moves in order from the move list generated by the function shown above, and then loops through the array to do the fitness calculation. (I.E. found and H at location [2,1] is Cord [1,1] [3,1] [2,0] or [2,2] also an H and if an H is found it just increments the count of bonds found)
After the calculation is complete the least fit chromosome is removed from my population and the new one is added and then the array list of chromosomes is sorted. Rinse and repeat until target solution is found
If you guys want to see more of my code to prove I actually did the work before asking for help just let me know (dont want to post to much so other students cant just copy pasta my stuff)
As suggested in the comments I have ran the profiler on my application (have never used it before, only a first year CS student) and my initial guess on where i am having issues was somewhat incorrect. It seems from what the profiler is telling me is that the big hotspots are:
When comparing the new chromosome to the others in the population to determine its position. I am doing this by implementing Comparable:
public int compareTo(Chromosome other) {
if(this.fitness >= other.fitness)
return 1;
if(this.fitness ==other.fitness )
return 0;
else
return -1;
}
The other area of issue described is in my actual evolution function, consuming about 40% of the CPU time. A codesample from said method below
double topPercentile = highestValue;
topPercentile = topPercentile * .20;
topPercentile = Math.ceil(topPercentile);
randomOne = numberGen.nextInt((int) topPercentile);
//Lower Bount for random two so it comes from outside of top 20%
int randomTwo = numberGen.nextInt(highestValue - (int) topPercentile);
randomTwo = randomTwo + 25;
//System.out.println("Selecting First: " + randomOne + " Selecting Second: " + randomTwo);
Chromosome firstChrom = (Chromosome) populationList.get(randomOne);
Chromosome secondChrom = (Chromosome) populationList.get(randomTwo);
//System.out.println("Selected 2 Chromosones Crossing Over");
Chromosome resultantChromosome = firstChrom.crossOver(secondChrom);
populationList.add(resultantChromosome);
Collections.sort(populationList);
populationList.remove(highestValue);
Chromosome bestResult = (Chromosome) populationList.get(0);
The other main preformance hit is the inital population seeding which is performed by the first code sample in the post

I believe the area in which I am taking the biggest hit is calculating the fitness of each Chromosome
If you are not sure then I assume you have not run a profiler on the program yet.
If you want to improve the performance, profiling is the first thing you should do.

Instead of repeatedly sorting your population, use a collection that maintains its contents already sorted. (e.g. TreeSet)

If your fitness measure is consistent across generations (i.e. not dependent on other members of the population) then I hope at least that you are storing that in the Chromosome object so you only calculate it once for each member of the population. With that in place you'd only be calculating fitness on the newly generated/assembled chromosome each iteration. Without more information on how fitness if calculated it's difficult to be able to offer any optimisations in that area.

Your random number generator seed doesn't need to be cryptographically strong.
Random numberGen = new Random();

A minor speedup when seeding your population is to remove all the testing and branching:
static Move[] moves = {Move.Down, Move.Up, Move.Left, Move.Right};
...
moveList.add(moves[randomMove]);

How to get facet ranges in solr results?

Assume that I have a field called price for the documents in Solr and I have that field faceted. I want to get the facets as ranges of values (eg: 0-100, 100-500, 500-1000, etc). How to do it?
I can specify the ranges beforehand, but I also want to know whether it is possible to calculate the ranges (say for 5 values) automatically based on the values in the documents?

To answer your first question, you can get facet ranges by using the the generic facet query support. Here's an example:
http://localhost:8983/solr/select?q=video&rows=0&facet=true&facet.query=price:[*+TO+500]&facet.query=price:[500+TO+*]
As for your second question (automatically suggesting facet ranges), that's not yet implemented. Some argue that this kind of querying would be best implemented on your application rather that letting Solr "guess" the best facet ranges.
Here are some discussions on the topic:
(Archived) https://web.archive.org/web/20100416235126/http://old.nabble.com/Re:-faceted-browsing-p3753053.html
(Archived) https://web.archive.org/web/20090430160232/http://www.nabble.com/Re:-Sorting-p6803791.html
(Archived) https://web.archive.org/web/20090504020754/http://www.nabble.com/Dynamically-calculated-range-facet-td11314725.html

I have worked out how to calculate sensible dynamic facets for product price ranges. The solution involves some pre-processing of documents and some post-processing of the query results, but it requires only one query to Solr, and should even work on old version of Solr like 1.4.
Round up prices before submission
First, before submitting the document, round up the the price to the nearest "nice round facet boundary" and store it in a "rounded_price" field. Users like their facets to look like "250-500" not "247-483", and rounding also means you get back hundreds of price facets not millions of them. With some effort the following code can be generalised to round nicely at any price scale:
public static decimal RoundPrice(decimal price)
{
if (price < 25)
return Math.Ceiling(price);
else if (price < 100)
return Math.Ceiling(price / 5) * 5;
else if (price < 250)
return Math.Ceiling(price / 10) * 10;
else if (price < 1000)
return Math.Ceiling(price / 25) * 25;
else if (price < 2500)
return Math.Ceiling(price / 100) * 100;
else if (price < 10000)
return Math.Ceiling(price / 250) * 250;
else if (price < 25000)
return Math.Ceiling(price / 1000) * 1000;
else if (price < 100000)
return Math.Ceiling(price / 2500) * 2500;
else
return Math.Ceiling(price / 5000) * 5000;
}
Permissible prices go 1,2,3,...,24,25,30,35,...,95,100,110,...,240,250,275,300,325,...,975,1000 and so forth.
Get all facets on rounded prices
Second, when submitting the query, request all facets on rounded prices sorted by price: facet.field=rounded_price. Thanks to the rounding, you'll get at most a few hundred facets back.
Combine adjacent facets into larger facets
Third, after you have the results, the user wants see only 3 to 7 facets, not hundreds of facets. So, combine adjacent facets into a few large facets (called "segments") trying to get a roughly equal number of documents in each segment. The following rather more complicated code does this, returning tuples of (start, end, count) suitable for performing range queries. The counts returned will be correct provided prices were been rounded up to the nearest boundary:
public static List<Tuple<string, string, int>> CombinePriceFacets(int nSegments, ICollection<KeyValuePair<string, int>> prices)
{
var ranges = new List<Tuple<string, string, int>>();
int productCount = prices.Sum(p => p.Value);
int productsRemaining = productCount;
if (nSegments < 2)
return ranges;
int segmentSize = productCount / nSegments;
string start = "*";
string end = "0";
int count = 0;
int totalCount = 0;
int segmentIdx = 1;
foreach (KeyValuePair<string, int> price in prices)
{
end = price.Key;
count += price.Value;
totalCount += price.Value;
productsRemaining -= price.Value;
if (totalCount >= segmentSize * segmentIdx)
{
ranges.Add(new Tuple<string, string, int>(start, end, count));
start = end;
count = 0;
segmentIdx += 1;
}
if (segmentIdx == nSegments)
{
ranges.Add(new Tuple<string, string, int>(start, "*", count + productsRemaining));
break;
}
}
return ranges;
}
Filter results by selected facet
Fourth, suppose ("250","500",38) was one of the resulting segments. If the user selects "$250 to $500" as a filter, simply do a filter query fq=price:[250 TO 500]

There may well be a better Solr-specific answer, but I work with straight Lucene, and since you're not getting much traction I'll take a stab. There, I'd create a populate a Filter with a FilteredQuery wrapping the original Query. Then I'd get a FieldCache for the field of interest. Enumerate the hits in the filter's bitset, and for each hit, you get the value of the field from the field cache, and add it to a SortedSet. When you've got all of the hits, divide the size of the set into the number of ranges you want (five to seven is a good number according the user interface guys), and rather than a single-valued constraint, your facets will be a range query with the lower and upper bounds of each of those subsets.
I'd recommend using some special-case logic for a small number of values; obviously, if you only have four distinct values, it doesn't make sense to try and make 5 range refinements out of them. Below a certain threshold (say 3*your ideal number of ranges), you just show the facets normally rather than ranges.

You can use solr facet ranges
http://wiki.apache.org/solr/SimpleFacetParameters#Facet_by_Range