We Keep Coding

Better to query once, then organize objects based on returned column value, or query twice with different conditions?

I have a table which I need to query, then organize the returned objects into two different lists based on a column value. I can either query the table once, retrieving the column by which I would differentiate the objects and arrange them by looping through the result set, or I can query twice with two different conditions and avoid the sorting process. Which method is generally better practice?
MY_TABLE
NAME AGE TYPE
John 25 A
Sarah 30 B
Rick 22 A
Susan 43 B
Either SELECT * FROM MY_TABLE, then sort in code based on returned types, or
SELECT NAME, AGE FROM MY_TABLE WHERE TYPE = 'A' followed by
SELECT NAME, AGE FROM MY_TABLE WHERE TYPE = 'B'

Logically, a DB query from a Java code will be more expensive than a loop within the code because querying the DB involves several steps such as connecting to DB, creating the SQL query, firing the query and getting the results back.
Besides, something can go wrong between firing the first and second query.
With an optimized single query and looping with the code, you can save a lot of time than firing two queries.
In your case, you can sort in the query itself if it helps:
SELECT * FROM MY_TABLE ORDER BY TYPE
In future if there are more types added to your table, you need not fire an additional query to retrieve it.

It is heavily dependant on the context. If each list is really huge, I would let the database to the hard part of the job with 2 queries. At the opposite, in a web application using a farm of application servers and a central database I would use one single query.
For the general use case, IMHO, I will save database resource because it is a current point of congestion and use only only query.
The only objective argument I can find is that the splitting of the list occurs in memory with a hyper simple algorithm and in a single JVM, where each query requires a bit of initialization and may involve disk access or loading of index pages.

In general, one query performs better.
Also, with issuing two queries you can potentially get inconsistent results (which may be fixed with higher transaction isolation level though ).
In any case I believe you still need to iterate through resultset (either directly or by using framework's methods that return collections).

From the database point of view, you optimally have exactly one statement that fetches exactly everything you need and nothing else. Therefore, your first option is better. But don't generalize that answer in way that makes you query more data than needed. It's a common mistake for beginners to select all rows from a table (no where clause) and do the filtering in code instead of letting the database do its job.

It also depends on your dataset volume, for instance if you have a large data set, doing a select * without any condition might take some time, but if you have an index on your 'TYPE' column, then adding a where clause will reduce the time taken to execute the query. If you are dealing with a small data set, then doing a select * followed with your logic in the java code is a better approach

There are four main bottlenecks involved in querying a database.
The query itself - how long the query takes to execute on the server depends on indexes, table sizes etc.
The data volume of the results - there could be hundreds of columns or huge fields and all this data must be serialised and transported across the network to your client.
The processing of the data - java must walk the query results gathering the data it wants.
Maintaining the query - it takes manpower to maintain queries, simple ones cost little but complex ones can be a nightmare.
By careful consideration it should be possible to work out a balance between all four of these factors - it is unlikely that you will get the right answer without doing so.

You can query by two conditions:
SELECT * FROM MY_TABLE WHERE TYPE = 'A' OR TYPE = 'B'
This will do both for you at once, and if you want them sorted, you could do the same, but just add an order by keyword:
SELECT * FROM MY_TABLE WHERE TYPE = 'A' OR TYPE = 'B' ORDER BY TYPE ASC
This will sort the results by type, in ascending order.
EDIT:
I didn't notice that originally you wanted two different lists. In that case, you could just do this query, and then find the index where the type changes from 'A' to 'B' and copy the data into two arrays.

Efficient way to check if record (from a large set of data) is existing in the Database (JPA/Hibernate)

We have a large set of data (bulk data) that needs to be checked if the record is existing in the database.
We are using SQL Server2012/JPA/Hibernate/Spring.
What would be an efficient or recommended way to check if a record exists in the database?
Our entity ProductCodes has the following fields:
private Integer productCodeId // this is the PK
private Integer refCode1 // ref code 1-5 has a unique constraint
private Integer refCode2
private Integer refCode3
private Integer refCode4
private Integer refCode5
... other fields
The service that we are creating will be given a file where each line is a combination of refCode1-5.
The task of the service is to check and report all lines in the file that are already existing in the database.
We are looking at approaching this in two ways.
Approach1: Usual approach.
Loop through each line and call the DAO to query the refCode1-5 if existing in the db.
//psuedo code
for each line in the file
call dao. pass the refCode1-5 to query
(select * from ProductCodes where refCode1=? and refCode2=? and refCode3=? and refCode4=? and refCode5=?
given a large list of lines to check, this might be inefficient since we will be invoking the DAO xxxx number of times. If the file say consists of 1000 lines to check, this will be 1000 connections to the DB
Approach2: Query all records in the DB approach
We will query all records in the DB
Create a hash map with concatenated refCode1-5 as keys
Loop though each line in the file validating against the hashmap
We think this is more efficient in terms of DB connection since it will not create 1000 connections to the DB. However, if the DB table has for example 5000 records, then hibernate/jpa will create 5000 entities in memory and probably crash the application
We are thinking of going for the first approach since refCode1-5 has a unique constraint and will benefit from the implicit index.
But is there a better way of approaching this problem aside from the first approach?

try something like a batch select statement for say 100 refCodes instead of doing a single select for each refCode.
construct a query like
select <what ever you want> from <table> where ref_code in (.....)
Construct the select projection in a way that not just gives you wnat you want but also the details of ref_code. Teh in code you can do a count or multi-threaded scan of resultset if DB said you got less refCodes that the number you codes you entered in query.

You can try to use the concat operator.
select <your cols> from <your table> where concat(refCode1, refCode2, refCode3, refCode4, refCode5) IN (<set of concatenation from your file>);
I think this will be quite efficient and it may be worth to try to see if pre-sorting the lines and playing with the num of concatenation taken each times bring you some benefits.

I would suggest you create a temp table in your application where all records from file are stored initially with batch save, and later you run a query joining new temp table and productCodes table to achieve filtering how you like. In this way you are not locking productCodes table many times to check individual row as SqlServer locks rows on select statement as well.

fetching records from database based on variable number of inputs from from

I have a form with 8 fields and based on the values entered in them I have have to fetch the records from the DataBase. Now the problem is out of the 8 fields the user may fill any number of fields and that too in any order for example the user may fill fields 1,4 and 6 or he may fill 1 and 7 or he may fill all of them (of course he has to fill at least one field)... Now how will I write a query which will work for any number and order of input parameters? and also because this query will be used in reporting(iReport) I am not allowed to write any code with it , it has to be a SQL query. Any ideas
Thanks

there are a number of ways to do this. I have a blog post about doing something like this in a Microsoft SQL (T-SQL) stored procedure at http://code.scottshipp.com/2013/03/29/tutorial-stored-procedures-with-truly-optional-parameters/ but it is likely that you will want to do something more complex and/or you are not using MS SQL Server. You may have to write the query fragments yourself. My suggestion is to do something like the following:
Once the form is submitted, loop through the various fields in the form, and check if their values are empty or not.
For those that are not empty, add a string containing an appropriate corresponding query fragment to some collections object, like an ArrayList. Do not include "AND" or "OR" directly in this string. If you need to keep track of whether this query fragment gets "AND"ed or "OR"ed with other query fragments, track that in a separate collections object. The "query fragment" I'm talking about is what would show up in the "where __" portion of your SQL query. For instance, say the search was for someone's last name. The query fragment you add is a string that says "lastName='" + lastNameField.value() + "'".
Once you have iterated through all the various fields in the form and have a final collections object full of query fragments, construct the final SQL statement from it. Iterate through your collections object (ArrayList in this example) and connect each one with the appropriate "AND" or "OR". Say your ArrayList has the three fragments "firstName='Joe'", "middleName='Q.'", "lastName='Public'". Use a StringBuilder to keep adding these fragments together into a final where clause: "firstName='Joe' AND middleName='Q.' AND lastName='Public'"...you may want to change these to a "LIKE" style query with wildcard characters.
You now have everything you need to create the final select statement. Issue it to the database and retrieve your results!

How to rearrange rows of resultset data in java?

List userProcessedCountCol = new ArrayList();
while (iResultSet1.next()) {
afRealTimeIssuance afRealTimeIssuance = new afRealTimeIssuance();
Integer i = 0;
afRealTimeIssuance.setSub_channel(iResultSet1.getString(++i));
afRealTimeIssuance.setAgent_names(iResultSet1.getString(++i));
afRealTimeIssuance.setFtd(iResultSet1.getDouble(++i));
afRealTimeIssuance.setMtd(iResultSet1.getDouble(++i));
afRealTimeIssuance.setQtd(iResultSet1.getDouble(++i));
userProcessedCountCol.add(afRealTimeIssuance);
}
where afRealTimeIssuance is ActionForm
Using the above snippet I get something like below output
1 A 100
2 B 200
3 C 300
4 D 400
But I want to rearrange the output as
2 B 200
4 D 400
3 C 300
1 A 100
In short I want to rearrange the rows as I want.How to arrange the resultset data based on one particular value.Please guide

you can act as at two levels here:
Database level
Java level
At the database level the only way to manipulate the order of results to be returned is using ''ORDER BY ASC/DESC'' in your sql query. Note, that you can't rely on any other way to get the ordered results from the database.
At the java level you can store your objects like this:
- use a sortable collection. Make your action form comparable or implement a comparator that
allows to sort elements as you wish.
Then your can use This method to get the ordered collection by your own criteria.
You can consider also using TreeSet instead of ArrayList
This data structure will allow you to just add the data and given the comparator that you've defined in advance it will be always sorted. The addition has a logarithmic complexity though, so its up to you to decide.
Hope this helps

The ResultSet cannot be rearranged manually (only with sql) . What you can rearrange is your data structure that you hold your Objects
You can use an ArrayList of your row Objects and insert each row in the position you would like.
Lets say in your example, in the while loop:
userProcessedCountCol.add(index, element);

There are two ways of doing this. One you can modify the query to use ORDER BY clause to arrange the results. Second you can implement the Comparator interface and define your comparator classes and use Collection.sort(List list,Comparator c) to order the data.

Either use an ORDER BY clause in your SQL query, or Collections.sort() the List using a Comparator<afRealTimeInssuance>. The former is easier and places the load on the database, the latter more versatile as you can sort based on external information.
On a side note, you should name your classes using the Java conventions: AFRealTimeInssuance instead of afRealTimeInssuance.

Unknown record number while executing SQLite query

When I am executing a SQLite query (using sqlite4java) I am following the general scheme of executing it step by step and obtaining one row at a time. My final result should be a 2-D array, whose length should correspond to the amount of records. The problem I am facing is the fact that I don't know in advance how many records are to be returned by my query. so I basically store them in an ArrayList and then copy pointers to the actual array. Is there a technique to somehow obtain the number of records to be returned by the query prior to executing it fully?

My final result should be a 2-D array, whose length should correspond to the amount of records.
Why? It would generally be a better idea to make the result a List<E> where E is some custom type representing "a record".
It sounds like you're already creating an ArrayList - so why do you need an actual array? The Collections API is generally more flexible and convenient than using arrays directly.

No, if using JDBC. You can, however, first do a COUNT() query, and then the real query.