Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
In my Application I build a Cache in the form of HashMap by taking its key-value from Database.
Now I have to add something in this cache, here what I am doing is update or insert this new value into the database and refresh Cache so it should populate by updated Database.
Now for that what I am doing is -
1. I am taking two Maps , cache1 and cache2 so when one cache is in use other will be an empty cache
2. When I want to refresh cache I start populating the empty cache and when it is done I clear the earlier cache.
3. All data will be given by this new cache. I just want to know if my approach is fine or I can do anything else which is more productive and efficient. Because I feel I don't need two map for doing this but then I think if I use only one it can affect the data accuracy.
EDIT :
Here is something you need to know about the application before answer.
1. The Data is neither so huge nor very small.
2. It takes approx 2-3 mins to populate cache from Database.
3. Refreshes are not much frequent only once or twice in a day required.
4. Application runs 24 Hrs(One Restart of Server/Day in early morning)
Something that you can try is EHCache, here you can find an example : ehcache
Also if you want to implement a simple cache you can use SoftReference. This is a much better way to implement cache than using HashMap.
As when you store a reference to HashMap it stores the Strong Reference to the object and when you set the object to null, it will still not be garbage collected as there is a strong reference to it in the map.
For more on caches here is a great link:
Cache based on SoftReference
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I have a ConcurrentHashMap which stores ID and timestamp when this ID was updated.
This data is to be persistent as application restart should be aware of the previous state of the data when shut-down was called.
I am currently serializing the map to a file on shut-down and loading it back when the application restarts.
However I can foresee that the serialization on shutdown would fail when the disk is full. And this would mean data-loss which is unacceptable.
I thought of using a DB to store the data but then it would add network weight on every update.
The only thing that comes to my mind right now is to serialize the map on every update. This would ensure that most of the data is persistent in case of disk full also even in case of Unexpected shutdown.
I am aware that this is a heavy operation and am open for alternative solutions.
Also note, this map may hold over 1200K entries...
Thanks in advance
If your scenario allowed some data loss then one solution can be
1.Periodically save snapshot of your hashmap so at most there will be a data loss for that interval.
2.For strict scenario you can log your action such that you can replay and get the original value.And as log is adding on end and read less may not be a performance hit.Log base technique used in like zookeper for meta data storage.
3.Or you can persist to some kind of db asynchronously by using queue and process in batch.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I am new to Android development and I am trying to make a Trivia application.
I need to store the data relating to questions somewhere and I am not entirely sure where to store it.
I plan to have multiple people playing so I need each person to have the same questions.
Basically I planned to have a list of categories and within each category I had question objects.
The question objects contained information regarding the question such as the answers and question itself.
However, if I use a database, I believe none of this would be needed due the questions being stored in tables which would represent categories.
In terms of speed what would be better:
to store it in a database
or to read from a file every time the application is loaded and store the data within a data structure?
You almost certainly want a database. Databases are made for fast search and easy insertion/deletion. There's really no advantage to having a file and doing in memory parsing each time.
Aside from performance benefits, here's a simple list of advantages of using SQLite rather than flat file:
You can query items as you wish -- don't need to load all of them and
select which ones you need.
Record deletion is a much less painful process. No rewriting of whole
files into wherever.
Updating a record is as easy as removing or creating one.
Have you ever tried doing cross-referencing lookups on a flat file?
Just.Not.Worth.It.
To summarize, it's every advantage a Database has over a text file.
Answer by josephus
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
I want to reduce the time of part of project that currently takes 2 hrs.
How it has been coded is it goes and take out almost 700,000 UID from one table and pass it to 16 different threads. Each thread then connect to JDBC and fetch a row for a UID one by one. it runs 700,000 query! 50k for each thread
Because it uses 3 to 4 fields of each row my plan is to get the needed fields at first and don't connect to database anymore.
my concerns:
because it fetch a row by UID ( I assume this should be fast) does it improve performance dramatically ?
I need to worry about memory and cache misses and everything, putting 700,000 rows with couple of fields in memory scares me.
Overall do you think this will help to improve the performance or you think it doesn't matter that much. saving 5min because of testing necessary it doesn't worth it.
So do you think I should pursue this path or focus more on logic???
Thanks a lot
As has been suggested in various comments, you should load the records in batches. I don't know how your infrastructure is configured, but if you are using a cloud service, a database round trip can take on the order of hundreds of milliseconds.
However, that may not be your only problem. If you haven't configured connection pooling, it may well be that you are gaining nothing from your multi-threading as each thread waits to grab the database connection. Your pool size should take into account how many connections may be established concurrently (in this case, it sounds like 17 may be your number - 1 for the main thread and 16 for the workers - if I understand your architecture correctly).
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I am going to write a game in which I have often have to check if a string of letters is actually a word or not. My question is about how to do this the fastest with the least computation-able power as possible (for instance an old smart-phone). With if possible not much start-up time to make it a quick and responsive app.
I once did this look-up by first reading in a word-file with almost all words into an appropriate sized hash-map of around 650,000 words*. (* might be more, I am not sure if this is the exhausted list yet).
Would a SQL database be appropriate here? I am thinking of buying a book about it so I can learn and implement one. Also I have no idea how you could create a hash-map, save it for later and then load one. Is that too much of a hacker solution or is that technique used more often? So would it make sense for me to learn SQL or do it with saving a hashmap and then later restoring it.
A database SQL could be appropriate if you plan to query it every time you need to check a word, but this is not the fastest solution; querying every single word slows down the response time but it should use less memory if the words number is high (you must measure the memory consumed by the db vs the memory consumed by the map). Checking if a word is inside a map is not so computationally expensive, it must calculate the hash and iterate over the array of items with the same hash.
Personally I would choose a map if the memory requirements of keeping all the words in memory can be satisfied. You can store the dictionary as plain text file (one line -> one word) and read it in a background thread when the application starts.
If memory is an issue, this seems like a good use for a B-Tree. This allows for O(log n) search time while searching a large amount of records with minimal memory usage. For this sort of application it sounds like loading the entire thing into memory is not going to be a good idea.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
So, I'll be looping through a data base and there will be a bunch of campaigns. Each campaign will have some demos and some sites where certain conditions are satisfied. I want to plot some graphs for the data corresponding to all the campaigns, sites and demos. I was thinking of using java, first getting the campaign, site and demo combinations where the conditions are satisfied and then looping through all of them, running individual queries based on their values and plotting the graphs using maybe, GNU plot. My questions are -
Is there a better way to achieve this (with minimal queries).
If I do do it this way, I first have to store the information.
I was thinking of storing the campaign ids in an ArrayList of Integers, the demos for each campaign in
ArrayList<ArrayList<Integer>>
and the sites for each campaign in
ArrayList<ArrayList<Integer>>
Is there a more efficient way of storing this information?
I'd recommend creating a new class to hold your campaign data and storing references to each object within an ArrayList if you need to keep a handle to them in memory (may not be necessary).
From a purist point of view, the class should be backed by a Data Access Object (DAO) and Plain Old Java Object (POJO) to manage database access and storage in memory but if this is a simple prototype then I wouldn't worry too much. I'd also recommend a utility class to convert/write your chart data - all accessible from your Campaign class.
The Campaign class should also be able to work out whether your conditions are satisfied - and if it's worth generating those charts.