Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I have a query which returns thousands of records (at some point it will). In that query I have some like this
Case when column in(:params1)
then :param2
when column in (:params3)
then :param4
when column in(:params5)
then :param6
when column in(:params7)
then :param8
END ABC
Now the question is what is better to do this in the query or return the column value and do the if\else in the pojo? And why? I tried testing it but currently don't have that much data.
Usually it is better (both because of performance and complexity) to let the database do as much work as possible for you. Doing the work in your application is likely to incur more network traffic than is necessary (which would decrease performance) and the code would have to contain all the nasty logic in it which would add complexity.
Also remember to avoid premature optimization. Try to avoid fixing problems that you don't have yet.
I would recommend letting the database do the work.
Returning thousands of records to the middle tier, operating on them, and shoving the result back into the database makes no sense to me. Why do all that network back and forth?
If you are truly processing that many records, I'd recommend considering letting the database do the work. No network traffic that way.
If not possible, you should make sure you truly need all those records. I'm betting you only think you do.
Writing queries this way seems like another bad idea to me.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed last year.
Improve this question
What is the Best way to convert Uppercase values to camel case.
Lets say I have API with response {"name":"VIRAT KOHLI","teams":"RCB INDIA DELHI"}
Should it be handle at DB query level or Java level (Business logic) or at Client-side like using Angular pipes. What is the best practices and what is best in terms of performance ?
At database I am using simple JDBC query.
So I don't think there's a huge difference in performance or speed.
However, if you are working with a very large amount of data, doing the conversion at the database level could add a little bit more time.
It all depends on your context but you can :
Do it by Angular using the "TitleCasePipe"(https://angular.io/api/common/TitleCasePipe#titlecasepipe)
or
avoid repetition by doing a ".map()" on the received data by putting in camelCase the concerned fields (there are a lot of examples out there).
Do it on the Business layer of your Backend just making sure to do the transformation before sending the response.
Finally do it from the database (not recommended) but you have to take into account that the code is always faster than a database query.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
Hello stackoverflow i have a doubt, which is the proper way to solve this?
I have this sql sentence:
SELECT *
FROM task
, subject
WHERE task.id_subject = task.id
AND task.id_tasktype = 1
AND subject.id_evaluation = {ALL the ids of table evaluation}
If i want to execute this sentence for every evaluation what is more efficient? a loop/cursor or whatever in SQL (i have basic knowledge of sql) or a regular for each in Java?
It depends on your situation. Basically, if your database server and application server are actually two different computers, then you might decide to run the loop at the server which can handle more pressure. You need to look at some statistics to be able to determine this.
Also, you can implement both solutions and measure the time needed at db server + time needed at application server. If one of your loops is consistently quicker than the other, then it is practically more efficient in the scenario you are running it according to your experiments. Off course, the scenario might change over time.
Generally speaking, people tend to run this loop on the application server (Java), since you might need to execute some things available only there in the future, but if you have a very good reason to run this on the database server, like the case when a trigger should trigger this functionality, then you might decide to run it there.
Basically, you are trying to optimize a loop where you do not necessarily have a problem. If you encounter performance issues, then you might decide to experiment with a few things, including, but not limited to the suggestions shown in your question.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
I want to reduce the time of part of project that currently takes 2 hrs.
How it has been coded is it goes and take out almost 700,000 UID from one table and pass it to 16 different threads. Each thread then connect to JDBC and fetch a row for a UID one by one. it runs 700,000 query! 50k for each thread
Because it uses 3 to 4 fields of each row my plan is to get the needed fields at first and don't connect to database anymore.
my concerns:
because it fetch a row by UID ( I assume this should be fast) does it improve performance dramatically ?
I need to worry about memory and cache misses and everything, putting 700,000 rows with couple of fields in memory scares me.
Overall do you think this will help to improve the performance or you think it doesn't matter that much. saving 5min because of testing necessary it doesn't worth it.
So do you think I should pursue this path or focus more on logic???
Thanks a lot
As has been suggested in various comments, you should load the records in batches. I don't know how your infrastructure is configured, but if you are using a cloud service, a database round trip can take on the order of hundreds of milliseconds.
However, that may not be your only problem. If you haven't configured connection pooling, it may well be that you are gaining nothing from your multi-threading as each thread waits to grab the database connection. Your pool size should take into account how many connections may be established concurrently (in this case, it sounds like 17 may be your number - 1 for the main thread and 16 for the workers - if I understand your architecture correctly).
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I have a three million rows in a database and I need to get all the values in a table as object and operate on those objects ? what is the best possible solution ?
The best solution is to not load all of them.
Why would need to load them all and operate on them?
Maybe you can do a SP (stored procedure) and work on these rows on the DB server.
If you still need to load them all, try to not load all columns of these rows.
Maybe you can use something like paging (if that is applicable to your case).
My answer is maybe too general but so is your question.
As Peter said, don't load them all. Instead, use an iterator, like a database cursor (ResultSet for the rows in a SQL query) to keep track of your place in the data. For any more specific answer, you'll need to give more detail, but you should also consider whether you can use SQL aggregation functions (COUNT, GROUP BY, etc.) to reduce the number of rows your application needs to process.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
What is the general guideline/rule for filtering data? I am accustomed to seeing filters in an SQL statement in the WHERE clause, although there are occasions that filters introduce complexity to the SQL, making it massive and abit hard to read on first glance for intermediate developers, but well-written ones that look complex are well-tuned and optimal. Filtering can also be done in Java, but that of course has a drawback where unfiltered data from SQL can be huge and loading it in memory only to filter it out may be wasteful. Of course there are cases where you have no choice but to filter in Java if you have several datasources as dependencies that the filter requires.
Filter on the backend (sql), whenever possible. If that makes the query too complex for a junior developer then so be it. While clarity of code is important, you shouldn't make design decisions based on how well a junior developer will understand it -- its sufficient that he be able to use it.
This is particularly the case when talking about different layers, your junior developer might not know any SQL, would you then avoid a SQL backend entirely?
Write your SQL to be as clear as possible (without sacrificing performance), but do so with the expectation that the person maintaining it will be familiar with SQL and how it should be used. When crossing layers like this, a little "easier to understand" can really kill your performance (pulling data back from the db in order to update it, can take thousands of times longer than a update on the DB, inappropriate use of a cursor can be expoentally worse than a set based solution).
If the data is already in the DB, then it makes more sense to do the filtering within it, since the RDBMS will be optimized for this kind of work. If the filtering can be confusing to novice and intermediate developers, why not hide it in a view, and only grant access to the view, to the users in question?
I guess there are no definite answers to it. It depends of individual use case. Both filtering in java as well as SQL can be applicable depending on the application under consideration.
As you mentioned, filtering in SQL can make the queries complex and and costly. But at the same this can be improved using database tuning, putting appropriate indexes, table partitioning etc. This is specially the case where in the database design is still evolving and you can do these type of changes.
if you are working on a system whose database is already designed and you have hardly any scope for significant changes (and hence not much query/database optimization), in this case filtering in java is better option.
It all depends on specific use case.