We Keep Coding

Retrieve and Insert million records into table

there's column I want to retrieve and insert into another table
For example, below is first table I want to retrieve values
Table1
Records
1 ABC Singapore
2 DEF Vietnam
I retrieve above column value from Table1, then insert into another table as below
Table 2
ID Name Country
1 ABC Singapore
2 DEF Vietname
Currently, I can do with java, I first retrieve records then split the values and insert. However, I want to do it by batch or pagination for better performance when Table1 got million of records to retrieve and insert those million records into Table2.
Any pointer to show me how to use pagination in my case would be appreciated.
I"m use MSSQL 2008

If you need to do that in code (and not in SQL which should be easier even with multiple delimiters), what you probably want to use would be batched inserts with proper batch size combined with a good fetch-size on your select:
//Prepare statements first
try(PreparedStatement select = con.prepareStatement("SELECT * FROM SOURCE_TABLE");
PreparedStatement insert = con.prepareStatement("INSERT INTO TARGET_TABLE(col1, col2, col3) VALUES (?,?,?)")) {
//Define Parameters for SELECT
select.setFetchDirection(ResultSet.FETCH_FORWARD);
select.setFetchSize(10000);
int rowCnt = 0;
try(ResultSet rs = select.executeQuery()) {
while(rs.next()) {
String row = rs.getString(1);
String[] split = row.split(" |\\$|\\*"); //However you want to do that
//Todo: Error handling for array length
//Todo: Type-Conversions, if target data is not a string type
insert.setString(1, split[0]);
insert.setString(2, split[1]);
insert.setString(3, split[2]);
insert.addBatch();
//Submit insert in batches of a good size:
if(++rowCnt % 10000 == 0) {
int[] success = insert.executeBatch();
//Todo: Check if that worked.
}
}
//Handle remaining inserts
int[] success = insert.executeBatch();
//Todo: Check if that worked.
}
} catch(SQLException e) {
//Handle your Exceptions
}
On calculating on "good" fetch and batch sizes you'll want to consider some parameters:
Fetchsize impacts memory consumption in your client. If you have enough of that you can make it big.
Committing an insert of millions of rows will take some time. Depending on your requirements you might want to commit the insert transaction every once in a while (every 250.000 inserts?)
Think about your transaction isolation: Make sure auto-commit is turned off as committing each insert will make most of the batching gains go away.

Returning a single column value from SQL database

I need to get the latest entry from my database, but not the autoincrement.
This function is in my databasehandler:
public int getLatestRouteNumber()
{
int number = 0;
SQLiteDatabase db = this.getReadableDatabase();
String query = "SELECT MAX("+ KEY_ROUTENUMBER + ") FROM " + TABLE_LOCATIONS;
Cursor c = db.rawQuery(query, null);
if (c.moveToFirst() && c != null) {
number = c.getInt(3);
}
return number;
}
it craches at the line where "number = c.getInt(3).
The third column in my database exists and has data in it.
The error I'm gettin is "Couldn't read row 0, col 3 from CursorWindow. Make sure the Cursor is initialized correctly before accessing data from it." I only need one value in the entire database, not even an entire row.

You have only one column in cursor but trying to get value of 3rd column... Tat is the error
Change it to getInt(0);

The returned result-set has nothing to do with table columns in your case. You should use
number = c.getInt(0);

Your resultset contains just one column and one row, basically a scalar value, it doesn't matter how many columns you have in your table, what you should consider here is the value you obtain from your query In your case just one value. Use
getInt(0)

have you tried something like this :
String query = "SELECT MAX("+ KEY_ROUTENUMBER + ") as my_max FROM " + TABLE_LOCATIONS;
And :
columnIndex = c.getColumnIndexOrThrow("my_max");
number = c.getInt(columnIndex );
That's better than using indexes, indexes may be wrong if the raw query changes.

getInt (3) returns the value of the 3rd column returned by the query. Your query only had 1 column.
Change getInt (3) to getInt (0)

"c.getInt(3);" does not get the value from the column in your table, it gets it from the column in your cursor.
And you only selected one column into your cursor.
So I believe it should be "c.getInt(1);"

Fetching single row result without iterating in loop from Mysql using Java

I have a simple query that returns only count of rows.
select count(*) cnt from table
I can read it by iterating through resultset.
like
while(rs.next()){
int rowCount= rs.getInt(cnt);
}
But is there any way,using which I can get count directly without looping.

How about:
int rowCount = rs.next() ? rs.getInt(cnt) : -1;
It doesn't save you much though

JPA: select random row

This is my JPA ENTITY
#Entity
#NamedQueries({
#NamedQuery(name = "Question.randQuestion", query = "SELECT q FROM Question AS q ORDER BY RANDOM")
})
#Table(name = "questions")
public class Question implements Serializable {
.....
}
The problem is:
eclipse gives me an error for this namedQuery. It says:
"The identification variable 'RANDOM' is not defined in the FROM clause"
I've tried also with RAND() instead of RANDOM and also NEWID().
Thanks.

To get a Random Row, first get list of total question and get any one.
public Question getRandomQuestion(EntityManager em) {
Query countQuery = em.createNativeQuery("select count(*) from Question");
long count = (Long)countQuery.getSingleResult();
Random random = new Random();
int number = random.nextInt((int)count);
Query selectQuery = em.createQuery("select q from Question q");
selectQuery.setFirstResult(number);
selectQuery.setMaxResults(1);
return (Question)selectQuery.getSingleResult();
}
Note: You may need to implement a logic to avoid duplicates while calling method more than once.

I had to solve a specific case of this problem where I had to select random records from a set of records matching a certain input criteria. The solution also had to support limit. I describe my solution below starting with assumptions.
Assumptions:
Given the set of criteria as input, it is possible to count number of records that match a selection criteria, as supported by the org.springframework.data.querydsl.QueryDslPredicateExecutor<T>.count(Predicate predicate) method.
Pages are zero indexed.
It is possible to request specific page as supported by the org.springframework.data.domain.PageRequest(int page, int size) method.
Algorithm
Count all records matching the input criteria.
Calculate total number of pages based on the count and specified limit.
Generate a random page index in range [0, total pages).
Request the page with index generated in previous step.
Shuffle elements in the returned page.
Code
Long totalRecords = someRepository.count(somePredicate);
Long totalPages =
(totalRecords % someLimit == 0)
? (totalRecords / someLimit)
: ((totalRecords / someLimit) + 1);
int pageIndex = (int) (Math.random() * totalPages);
PageRequest pageRequest = new PageRequest(pageIndex, someLimit);
Page<T> somePage = someRepository.findAll(somePredicate, pageRequest);
List<T> someList;
if (somePage.getTotalElements() > 0) {
someList = new ArrayList<>(somePage.getContent());
} else {
someList = new ArrayList<>();
}
Collections.shuffle(someList);
The second shuffle is to ensure records within the page are also randomized. The general case of this solution is that there is no criteria and so the count() has to be invoked with no predicate thus getting a count of all rows in the table.

As far as I understand you want to select random question from the table. You'd rather use WHERE clause, with providing some parameter from your code, like:
SELECT q FROM Question AS q WHERE id = :id
Then in the code, which creates your query you must generate random id to be selected:
query.setParam("id", getRandomId());
And to get random id, you may want to query number of rows from DB and use java.util.Random.nextInt(rowsCount) (if all ids are there, of course).
Btw, something similar is described here: http://www.shredzone.de/cilla/page/53/how-to-fetch-a-random-entry-with-hibernate.html

how to sort a column of a table in database with hibernate

I want to find 10 nearset value of a column of a table in a database to my value.
so I want to sort the value of that column, and then find 10 smaller or bigger value than my value.
how can I do this
thanks a lot for your help

HQL supports ORDER BY.
Either you do
Query q = session.createQuery("from Table order by abs(value - :v) asc";
q.setXxx("v", myValue); /* Xxx is Float or Long or Integer or... */
q.setMaxResults(10);
List<Table> l = q.list();
or
Query q1 = session.createQuery("from Table where value >= :v order by value asc";
q1.setXxx("v", myValue); /* Xxx is Float or Long or Integer or... */
q1.setMaxResults(10);
List<Table> l1 = q1.list();
Query q2 = session.createQuery("from Table where value < :v order by value desc";
q2.setXxx("v", myValue); /* Xxx is Float or Long or Integer or... */
q2.setMaxResults(10);
List<Table> l2 = q2.list();
/* now find the 10 nearest elements in Java code */
...
while (...) {
...
}
In the second example you have the inconvenience of two selects which give you 20 rows altogether and then you have to find the 10 nearest in Java code, but if there is a database index on the value column it might b a lot faster. The result will be the same for both examples.