Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I have a ConcurrentHashMap which stores ID and timestamp when this ID was updated.
This data is to be persistent as application restart should be aware of the previous state of the data when shut-down was called.
I am currently serializing the map to a file on shut-down and loading it back when the application restarts.
However I can foresee that the serialization on shutdown would fail when the disk is full. And this would mean data-loss which is unacceptable.
I thought of using a DB to store the data but then it would add network weight on every update.
The only thing that comes to my mind right now is to serialize the map on every update. This would ensure that most of the data is persistent in case of disk full also even in case of Unexpected shutdown.
I am aware that this is a heavy operation and am open for alternative solutions.
Also note, this map may hold over 1200K entries...
Thanks in advance
If your scenario allowed some data loss then one solution can be
1.Periodically save snapshot of your hashmap so at most there will be a data loss for that interval.
2.For strict scenario you can log your action such that you can replay and get the original value.And as log is adding on end and read less may not be a performance hit.Log base technique used in like zookeper for meta data storage.
3.Or you can persist to some kind of db asynchronously by using queue and process in batch.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I'm new to using a database, specifically MySQL. I'm creating a web application for class in which you can look up the name of a book and it'll display the summary of the book. My question is should I send a query to the database that collects all of the books' data on initialization and put them into a HashMap inside a manager class for lookup or should I use a query each time to lookup a specific book information?
It depends on the data transport time I would say. If your average query time times the number of request goes faster than a script to put everything into a HashMap, use queries. Otherwise, use a script that collects everything and puts it into a HashMap.
But if you have thousands of rows, you should use queries, because otherwise you will use too much RAM.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I am new to Android development and I am trying to make a Trivia application.
I need to store the data relating to questions somewhere and I am not entirely sure where to store it.
I plan to have multiple people playing so I need each person to have the same questions.
Basically I planned to have a list of categories and within each category I had question objects.
The question objects contained information regarding the question such as the answers and question itself.
However, if I use a database, I believe none of this would be needed due the questions being stored in tables which would represent categories.
In terms of speed what would be better:
to store it in a database
or to read from a file every time the application is loaded and store the data within a data structure?
You almost certainly want a database. Databases are made for fast search and easy insertion/deletion. There's really no advantage to having a file and doing in memory parsing each time.
Aside from performance benefits, here's a simple list of advantages of using SQLite rather than flat file:
You can query items as you wish -- don't need to load all of them and
select which ones you need.
Record deletion is a much less painful process. No rewriting of whole
files into wherever.
Updating a record is as easy as removing or creating one.
Have you ever tried doing cross-referencing lookups on a flat file?
Just.Not.Worth.It.
To summarize, it's every advantage a Database has over a text file.
Answer by josephus
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
I want to reduce the time of part of project that currently takes 2 hrs.
How it has been coded is it goes and take out almost 700,000 UID from one table and pass it to 16 different threads. Each thread then connect to JDBC and fetch a row for a UID one by one. it runs 700,000 query! 50k for each thread
Because it uses 3 to 4 fields of each row my plan is to get the needed fields at first and don't connect to database anymore.
my concerns:
because it fetch a row by UID ( I assume this should be fast) does it improve performance dramatically ?
I need to worry about memory and cache misses and everything, putting 700,000 rows with couple of fields in memory scares me.
Overall do you think this will help to improve the performance or you think it doesn't matter that much. saving 5min because of testing necessary it doesn't worth it.
So do you think I should pursue this path or focus more on logic???
Thanks a lot
As has been suggested in various comments, you should load the records in batches. I don't know how your infrastructure is configured, but if you are using a cloud service, a database round trip can take on the order of hundreds of milliseconds.
However, that may not be your only problem. If you haven't configured connection pooling, it may well be that you are gaining nothing from your multi-threading as each thread waits to grab the database connection. Your pool size should take into account how many connections may be established concurrently (in this case, it sounds like 17 may be your number - 1 for the main thread and 16 for the workers - if I understand your architecture correctly).
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
for some test data scenario i need to read file containing 100000 lines of row and process each row with some condition and then based on condition need to output the data in text format
for this i am planning to store all lines of data in some pojo then pojo to List
My worry is to having pojo of 100000 row in memory . this is just for testing case .
I think using InputSteam to read the file will be better since you still fetch rows one by one. You can read one line per time, and process your confition and then output.
Storing too much Objects in List may encounter an Out of Memory Error.
In any case, its a bad design to store all 100000 rows as POJO in memory. Some of the possible solutions are:
Read one row at a time and process it.
Rather than reading from a file one record at a time and processing it using java, use some scripting language to populate a database table, and then from your java code you can process the records from the table.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
In my Application I build a Cache in the form of HashMap by taking its key-value from Database.
Now I have to add something in this cache, here what I am doing is update or insert this new value into the database and refresh Cache so it should populate by updated Database.
Now for that what I am doing is -
1. I am taking two Maps , cache1 and cache2 so when one cache is in use other will be an empty cache
2. When I want to refresh cache I start populating the empty cache and when it is done I clear the earlier cache.
3. All data will be given by this new cache. I just want to know if my approach is fine or I can do anything else which is more productive and efficient. Because I feel I don't need two map for doing this but then I think if I use only one it can affect the data accuracy.
EDIT :
Here is something you need to know about the application before answer.
1. The Data is neither so huge nor very small.
2. It takes approx 2-3 mins to populate cache from Database.
3. Refreshes are not much frequent only once or twice in a day required.
4. Application runs 24 Hrs(One Restart of Server/Day in early morning)
Something that you can try is EHCache, here you can find an example : ehcache
Also if you want to implement a simple cache you can use SoftReference. This is a much better way to implement cache than using HashMap.
As when you store a reference to HashMap it stores the Strong Reference to the object and when you set the object to null, it will still not be garbage collected as there is a strong reference to it in the map.
For more on caches here is a great link:
Cache based on SoftReference