Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I have a three million rows in a database and I need to get all the values in a table as object and operate on those objects ? what is the best possible solution ?
The best solution is to not load all of them.
Why would need to load them all and operate on them?
Maybe you can do a SP (stored procedure) and work on these rows on the DB server.
If you still need to load them all, try to not load all columns of these rows.
Maybe you can use something like paging (if that is applicable to your case).
My answer is maybe too general but so is your question.
As Peter said, don't load them all. Instead, use an iterator, like a database cursor (ResultSet for the rows in a SQL query) to keep track of your place in the data. For any more specific answer, you'll need to give more detail, but you should also consider whether you can use SQL aggregation functions (COUNT, GROUP BY, etc.) to reduce the number of rows your application needs to process.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
I need a very lightweight and persistent key/value-store in Java.
The amount of data is very very low and it should be very simple (getter and setter and all can operate on strings).
So I think of using some small NoSQL-DB or even giving some integrated collection a serializer/deserializer to the filesystem.
But I think NoSQL is a overkill and I hope a persister also exists for such a simple requirement.
Whats the best approach here? Any ideas?
You can either implement your own thing if it is a simple key-value string. (Have a look at Java's Properties class too in case it suits your requirements).
If your requirements are slightly more complex have a look at the embedded lightweight databases you can use. Maybe BerkleyDB might work for you. There are quite a number of others if you do a bit of search.
Also think about what you actually need to do with the data. Do you need to query it (so it needs to be indexed?) or do you just want to load it back all into memory? (in which case using a simple JSON or YAML text format would also suffice.)
Most Map<String,String> can be serialized. So for example look into https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html
there you find Serializable. Under that point information to help yourself solve the Problem.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I'm new to using a database, specifically MySQL. I'm creating a web application for class in which you can look up the name of a book and it'll display the summary of the book. My question is should I send a query to the database that collects all of the books' data on initialization and put them into a HashMap inside a manager class for lookup or should I use a query each time to lookup a specific book information?
It depends on the data transport time I would say. If your average query time times the number of request goes faster than a script to put everything into a HashMap, use queries. Otherwise, use a script that collects everything and puts it into a HashMap.
But if you have thousands of rows, you should use queries, because otherwise you will use too much RAM.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I am new to Android development and I am trying to make a Trivia application.
I need to store the data relating to questions somewhere and I am not entirely sure where to store it.
I plan to have multiple people playing so I need each person to have the same questions.
Basically I planned to have a list of categories and within each category I had question objects.
The question objects contained information regarding the question such as the answers and question itself.
However, if I use a database, I believe none of this would be needed due the questions being stored in tables which would represent categories.
In terms of speed what would be better:
to store it in a database
or to read from a file every time the application is loaded and store the data within a data structure?
You almost certainly want a database. Databases are made for fast search and easy insertion/deletion. There's really no advantage to having a file and doing in memory parsing each time.
Aside from performance benefits, here's a simple list of advantages of using SQLite rather than flat file:
You can query items as you wish -- don't need to load all of them and
select which ones you need.
Record deletion is a much less painful process. No rewriting of whole
files into wherever.
Updating a record is as easy as removing or creating one.
Have you ever tried doing cross-referencing lookups on a flat file?
Just.Not.Worth.It.
To summarize, it's every advantage a Database has over a text file.
Answer by josephus
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I have a query which returns thousands of records (at some point it will). In that query I have some like this
Case when column in(:params1)
then :param2
when column in (:params3)
then :param4
when column in(:params5)
then :param6
when column in(:params7)
then :param8
END ABC
Now the question is what is better to do this in the query or return the column value and do the if\else in the pojo? And why? I tried testing it but currently don't have that much data.
Usually it is better (both because of performance and complexity) to let the database do as much work as possible for you. Doing the work in your application is likely to incur more network traffic than is necessary (which would decrease performance) and the code would have to contain all the nasty logic in it which would add complexity.
Also remember to avoid premature optimization. Try to avoid fixing problems that you don't have yet.
I would recommend letting the database do the work.
Returning thousands of records to the middle tier, operating on them, and shoving the result back into the database makes no sense to me. Why do all that network back and forth?
If you are truly processing that many records, I'd recommend considering letting the database do the work. No network traffic that way.
If not possible, you should make sure you truly need all those records. I'm betting you only think you do.
Writing queries this way seems like another bad idea to me.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
for some test data scenario i need to read file containing 100000 lines of row and process each row with some condition and then based on condition need to output the data in text format
for this i am planning to store all lines of data in some pojo then pojo to List
My worry is to having pojo of 100000 row in memory . this is just for testing case .
I think using InputSteam to read the file will be better since you still fetch rows one by one. You can read one line per time, and process your confition and then output.
Storing too much Objects in List may encounter an Out of Memory Error.
In any case, its a bad design to store all 100000 rows as POJO in memory. Some of the possible solutions are:
Read one row at a time and process it.
Rather than reading from a file one record at a time and processing it using java, use some scripting language to populate a database table, and then from your java code you can process the records from the table.