We Keep Coding

How does terraria have so big worlds and still has silky smooth fps?

I was working on my own procedural generator and finished making it.
Here is how I did it,
generate the entire world at the beginning,
store info about spawn positions and tiles positions into arrays and data structures
Now obviously, I have to loop through all the arrays, to render/update stuff,
this is creating problems,
I tried dealing with this problem by applying a condition that if any object is visible, only then render update else don't, but it does not make much of a difference.
my TILE_SIZE is 1 world units
so when my world size was
a)500 by 500,fps was 60
b)2000 by 2000, fps was 50
c)5000 by 5000 fps decreased further and so on
If only I could know how Terraria does it, or any way it done, would really help thanks.

I have to loop through all the arrays, to render/update stuff
You don't have to do that.
For rendering stuff you really only need to consider the blocks that are close to the player. Here is some code(x, y are the player coordinates, width, height are the dimensions of your world, render(x, y) is a placeholder for your rendering functionality, and for simplicity I assumed the player can see 100 blocks in each direction):
for (int i = Math.max(x-100, 0); i ≤ Math.min(x+100, width); i++) {
for (int j = Math.max(y-100, 0); j ≤ Math.min(y+100, height); j++) {
render(i, j);
}
}
The process of updating can also be simplified:
For npcs you only need to check a few blocks around it.
For block updates(liquid flowing, sand falling, corruption spread, …) there are actually two possibilities:
Iterate through a different part of the array each update:
Block updates don't have to be that frequent. For fluid flow it might be good enough to update only every 50th block in each update.
You can store the blocks that might need to be updated in an ArrayList(I suggest storing only the coordinates, not the blocks itself).
Then when the update function is called you go through that list and check wether the block really needs an update. If that's the case, you add the blocks neighbors to the list, which will then get updated in the next iteration.
Make sure to remove the blocks you just updated from the list and avoid adding duplicates!
While the second approach is normally faster(ususally the list is only determined by player actions and even if the player decides to drain the entire ocean the whole list would probably take less than ¹⁄₂₀ of the entire world).
But the first approach is certainly easier to implement and will be faster if for whatever reason you have lots of block updates.

Anything you do not see on screen is stored as values. This means that all tiles that are not visible in your viewbox do not get rendered. They are stored as objects (e.g. {x, y, type}); Make sure you count how many tiles are on screen VS how many you are rendering each loop in your code. Never render something that is not visible to the user as this will eat up your memory.
Any tile groups that cannot be rendered soon (i.e. by moving left/right) are not looped through. So say you are using tile sectors to group your tiles, and you're in sector D. Moving left will put you in C, and moving right will put you in E. So these sectors should be loaded in to arrays (but not rendered) There is no need for you to loop through any other sectors (e.g. A or B). So these can be stored externally (e.g. as text files). So they are available, but not taking up space in your games memory.
Good luck. :)

JavaFX (8) - Improving scatter chart performance

Question:
In what ways can I improve render-time in scenario described below?
Description:
The task I'm facing is plotting quite a few points on a scatter chart in JavaFX. I'm getting the data from an Oracle DB.
I think my current bottleneck is scatterChart.getData().add(series);
The series contains all the datapoints needed in the chart.
Executing this, and rendering the chart on screen may take anywhere from several seconds to several minutes. During this time the GUI freezes up.
We're running some Lenovo Thinkpads with 4 Cores 4 Threads, Intel HD4000 Graphics.
Some examples of finished charts are below. The latter being the current worst case, but as more data is collected, more data will be displayed.
(due to images being impractically large, only links)
Chart with ~5.5K points
Chart with ~75K points
More info
I followed this tutorial because I'm new to JavaFX, so I'm not sure exactly how rendering happens.
Code sample
This isn't the actual code I'm using, it's just to illustrate how I'm doing things.
//Reference to the chart in the fxml file.
#FXML
private ScatterChart<Date, Number> scatterChart
= new ScatterChart<>(new DateAxis(), new NumberAxis());
private void handleSelectionUpdate(PerfResultSet newValue){
//newValue is an object containing all the data from the database
XYChart.Series<Date, Number> series = new XYChart.Series<>();
/**
* adding all the data from newValue to series as
* XYChart.Data<Date, Number>
*
* this is handled by multiple threads and does not take long
*/
series.setName("myName");
//this statement is where the gui freezes
//can I optimize this in any way?
scatterChart.getData().add(series);
}

I faced similar problem - I was drawing route consisting of many points to google map with Javascript. I eliminated points taking to account current scale. I saw your png files, it seems you really need ~30-50 points for expressing the following message - hey, this vertical/horizontal segment contains so many points you may assume it solid line!
Next problem is max scale - I think, each solid line in 75K chart is actually not solid. I mean, in small scale, I see a lot points like solid line ____ but in max scale I see them as . . . ... . .. But usually, max scale required only for small piece of chart and you may eliminate not by scale but by borders. Again reference to google map - when you zoom the map, you are not seeing whole globe in cool resolution but some small piece of map. Other borders eliminated automatically.
Post here your code, some information about scale, zoom (do you have such feature?). If you haven't and your chart is always as in png files in link, you should iterate over each vertical or horizontal line, look for sequental points and replace, for example, 100 sequantal points to middle point of that 100 points. Instead of 100 experimentally find such X, that replacing sequental X points to one middle-value-point does not affect the resulting chart.
I repeat the main idea - There exist some X, such that drawing X sequental points equal to drawing 1 point in some fixed scale - many points does not give precision but simply "overwrites" each other due to scale and human eye limitations.
Some pseudocode for finding X if your scale is constant;
int MAGIC = 50; //change it on each run of your program! You have to find the best value
int counter=1; // points[0] taken as included in accumulating solid segment
int startIndex=0,endIndex;
ArrayList<Integer> compressedDatesArray = new ArrayList<Integer>();
for(int i=1;i<datesArray.length;i++){
if(deltaBetweenDates(datesArray[i]-datesArray[i-1])==1){
counter++;
endIndex=i;
if(counter==MAGIC){
counter=0;
compressedDatesArray.add(datesArray[{endIndex-startIndex)/2]);
startIndex=endIndex+1;
}
}
}
Render compressedArray, see png. If result is bad - chnage MAGIC and repeat
UPDATE
You may binary search MAGIC - try 5000, if still solid after comperssing 5000 points to one, try bigger value if not, try 2500 and so on... Humans also may binary search something=)))
Also, do not forget to remember Y value of point representing big segment. I mean, also declare compressedY and add correspinding Y of datesArray[(endIndex-startIndex/2)]

Tiled Terrain Generation with Perlin Noise - Java

I'm currently generating 2d Perlin noise to a 2d array and (after interpolation) rendering the results held as a height map (essentially array[x][z] = y).
This is fine for one array, but not for tile loading based on the camera position, and I'm having difficulty where the tiles should meet at the seam.
[Noise Tiles]
My best attempt has been to generate a large 2d array of the base noise (-1 to 1), then have each tile as an object that stores an offset value for the base grid (which part of the noise to read).
This has allowed me to sample (for interpolation) areas of the base grid that are much larger than the tile array, but it still not lining up!
My objective is to have a flycam that loads tiles as it moves, and can travel very far without repeating.
Is this approach of 2d arrays solid enough to handle the objective?
If so, how can it be implemented to tile or wrap correctly?

I have had similar problems before and after alot of tweaking and testing I've come to the conclusion that just plain 2D perlin noise as is will never look like natural terrain, it's essentially white noise(ie no huge mountains or valleys, just hills close together)
What I recently found as the way to go is by making a fractal out of multiple Perlin Noises but with different "resolutions" -if you will, added together to get custom terrain resolution using different octaves(ie higher octave = higher noise in your terrain). I usually go about using gain = 0.5(this really do not have to be changed much, it looks pretty good as is), and octaves = ~16
Note; this is made in processing, Java might have some different syntax, but it should be quite the same
float fractal(int x,int y,float gridSubs,float gain,int octaves){
float total = 0;
float freq = 1.0/gridSubs;
float amp = gain;
for(int f=0;f<octaves;f++){
total += noise(x*freq,y*freq)*amp;
freq *= 2.0;
amp *= gain;
}
return total;
}
Website where I found it: here
If you replace your noise() function with this you might get better looking results in the end.
As for your problem with your seams you probably have the coordinate offset in the noise function call for each chunk set wrongly, it should look somewhat like this:
noise( localX * tileSize + (chunkX * chunkSize) , localZ * tileSize + (chunkZ * chunkSize) );
You might have to add some resolution koefficent to make the noise somewhat smooth.
Now You said you are storing the heightvalues in a 2D heightMap, now that is fine and makes the heightvalues easy to access if you need to update them often or need to access them easily.
The problem with this is the size of the array easily get very large, and by that I mean, Very large. With my past experiences I could get a small map(can't remember size, but smaller than yours) to eat up 4Gb of my RAM just by loading it. I did use a float array though so using an integer array could have reduced the memory usage slightly.
How I do it now is I just recalculate each point in the terrain whenever I need to update the geometry, how I currently have set it up it does introduce a slight lagspike each time the terrain is changed when moving around. The benefit is that I can have a 4times larger map with greater detail and still use about 1-2Gb of RAM.
Since what I understood, you simply want to move around the terrain and look at it. This would benefit from not storing the heightmap in an array, since you do not really need the values after you have generated the terrain(this may differ if you are manually smoothing out the terrain).
One last thing to note; I am not an professional programmer, this is just what I have learned from my past experiences using randomly generated noise.

Implementing a tiling system in Android

I'm a bit stumped on an Android game I am developing, so maybe someone can help me. Don't bother suggesting AndEngine; I've made it this far on my own and I'd prefer to continue on this path if possible.
The game I'm working on is a simple platformer. Originally the levels were loaded as massive bitmaps, but a few out of memory errors made it obvious that I had to come up with a more efficient system. Now I'm parsing out .tmx files into a 2d array and using a method that creates a bitmap slightly bigger than the device's screen and fills it with the appropriate tiles.
The issue is that accessing this 2d array 1500 times (using 16x16 tiles) is very expensive, and I'm having trouble figuring out a way to not do that so much. Currently I'm having to redraw the image every time I pass over 16 pixels. Current ideas include using larger tiles (smaller array, still limits level size though), attempting to shift the contents of my bitmap and only load in the new row about to be displayed, or using a larger bitmap so that new images don't need to be loaded as often. Any suggestions?

Unless already done or impossible; consider separating the background from the foreground.
If you use a simple background (think Super Mario plain blue backgrounds) then you can most likely reuse those tiles/that bitmap/color/gradient much longer than the foreground elements.
This will also make your array/s contain lots of empty spaces where no foreground exists, meaning you could collapse the array and only store the parts containing tiles. Collapsing the arrays will come with the drawback that you will have to store a height attribute for the tiles to know where they are supposed to go.
(A background separate from the foreground also give the possibility of parallax-effects (background moving in different speed than foreground))
An example of how this changes your lookup:
Consider the game is being run on a 480x800 screen in landscape-mode.
This means 50*30 (1500) tiles to look up.
Every tile is saved in a 2d-array.
The current visible area looks like this:
A ground area, 4 tiles high.
A platform, 5 tiles high and 20 tiles wide, somewhere above the ground.
Another platform 3 tiles high and 5 tiles wide, also above ground.
A wall extending from the ground to the top of the screen, 5 tiles wide.
Removing the background tiles and collapsing the arrays we only have to look up the tiles:
Ground tiles: 50*4
Platform1 tiles: 5*20
Platform2 tiles: 3*5
Wall tiles: 26*5
Total lookups: 50*4 + 5*20 + 3*5 + 26*5 = 445 which is less than a third of the original 1500.
Now, of course there is some extra work to be done to figure out the position of each tile and drawing the background, but assuming a badly optimized case where this takes as much time as the lookup it is still less time than 1000 lookups.
Of course, this is just one of possibly many approaches to the problem.
Also remember small things like that 2d arrays are much slower than 'flat' arrays. If you cannot put the data in a 1d-array, try to minimize the nested calls, e.g. get one row or column at a time and perform the lookups from that array before switching to the next:
//Really bad:
for(int x=0; x<array.length; x++){
for(int y=0; y<array[x].length; y++){
Object o = array[x][y];
}
}
//Better (can be made even better by using the for(Object o : array)-syntax):
for(int x=0; x<array.length; x++){
Object[] yArray = array[x];
for(int y=0; y<yArray.length; y++){
Object o = yArray[y];
}
}

I had to develop this project where we needed to download tiles of image from the web and display them as one whole image. The thing was a casual scroll would end up crashing the entire app because the VM would throw a OutOfMemory Exception. So I developed a view using a TwoDScrollView and MxN ImageViews, which downloaded only those views which were in the users view. As the user scrolls, we maintain a cache and start de allocating images that are "far" away from the viewing region. It was slower but it stopped crashing.
I won't be able to share my part of the code. But here's a Github repo that uses the same approach. Good Luck :)
I can't vouch for them but here's a project specifically fits your needs

Heatmap with 10000 * 400 points in java

I want to show users a heatmap with 10000 * 400 tiles. Current state in my program is that I have a class which is a JComponent. In this JComponent, I have instances of Image for the tiles. If I had a 300 * 300 matrix, the program uses between 800 and 1000 MB ram. This is too much.
What can I do else or has someone a control which can show such a big heatmap?
Today I have tested JHeatChart, which creates a BufferedImage. There the size for a 10000 * 400 matrix is under 400 MB, but the class needs over 15 Minutes to create it. This is too long.
Has anybody some idee or knows an control which can handle the data?

Since this will be a map, you do not need to create a component for each tile. many of them can be re-used. What you want to do is during initialization of the app, create a map of all possible components, and then on each tile pass a reference to that component.
Here would be an example.
public class HeatMap{
private static List<JComponent> tiles = new ArrayList<JComponent>();
private List<JComponent> heatmap = new ArrayList<JComponent>();
private Random rand = new Random();
static{
tiles.add(new JLabel("Cold"));
tiles.add(new JLabel("Hot"));
tiles.add(new JLabel("Warm"));
}
public HeatMap(){
for(int i=0; i<10000; i++){
for(int j=0; j<400; j++){
heatmap.add(tiles.get(rand.nextInt(3)));
}
}
}
}
In the above case we only have to create 3 tiles, the rest are references to those. This type of approach should help reduce your memory usage.

You could try scaling your map. So instead of priniting every single tile to your screen, if your zoomed far away, use bigger tiles. One could also try to merging the single tiles to one object instead of keeping references for every single tile.

Obviously, 10000 * 400 tiles is too many to view at once. Instead, use the fly-weight pattern to render only the tiles that are visible in the viewport of a JScrollPane. JTable is an example.

Does a tile really have to be represented as an image or would a colored box be sufficient?
If a colored box would be sufficient you might just draw the visible tiles in your paint(...) method. There would be no need to load those images.
If it has to be images, I assume that not every tile is unique, thus instead of loading an image per tile you might try and load all needed images once and have the tiles reference the corresponding image. Thus if you have 100 different images, instead of having 4000000 copies you'd have 100 and 4000000 quite small references.

Any way you present it, the user will never be able to perceive four million data points simultaneously. Therefore you should never have four million graphical widgets. Either the display should be scaled down, or (when zoomed in) you should display only a subset of all tiles at a time. Either way you will have to map the notional 4,000,000 to far fewer graphical widgets and reuse those you already have.
As other answers have specified, you should also look into using the simplest possible widget type that will do the job.