there's column I want to retrieve and insert into another table
For example, below is first table I want to retrieve values
Table1
Records
1 ABC Singapore
2 DEF Vietnam
I retrieve above column value from Table1, then insert into another table as below
Table 2
ID Name Country
1 ABC Singapore
2 DEF Vietname
Currently, I can do with java, I first retrieve records then split the values and insert. However, I want to do it by batch or pagination for better performance when Table1 got million of records to retrieve and insert those million records into Table2.
Any pointer to show me how to use pagination in my case would be appreciated.
I"m use MSSQL 2008
If you need to do that in code (and not in SQL which should be easier even with multiple delimiters), what you probably want to use would be batched inserts with proper batch size combined with a good fetch-size on your select:
//Prepare statements first
try(PreparedStatement select = con.prepareStatement("SELECT * FROM SOURCE_TABLE");
PreparedStatement insert = con.prepareStatement("INSERT INTO TARGET_TABLE(col1, col2, col3) VALUES (?,?,?)")) {
//Define Parameters for SELECT
select.setFetchDirection(ResultSet.FETCH_FORWARD);
select.setFetchSize(10000);
int rowCnt = 0;
try(ResultSet rs = select.executeQuery()) {
while(rs.next()) {
String row = rs.getString(1);
String[] split = row.split(" |\\$|\\*"); //However you want to do that
//Todo: Error handling for array length
//Todo: Type-Conversions, if target data is not a string type
insert.setString(1, split[0]);
insert.setString(2, split[1]);
insert.setString(3, split[2]);
insert.addBatch();
//Submit insert in batches of a good size:
if(++rowCnt % 10000 == 0) {
int[] success = insert.executeBatch();
//Todo: Check if that worked.
}
}
//Handle remaining inserts
int[] success = insert.executeBatch();
//Todo: Check if that worked.
}
} catch(SQLException e) {
//Handle your Exceptions
}
On calculating on "good" fetch and batch sizes you'll want to consider some parameters:
Fetchsize impacts memory consumption in your client. If you have enough of that you can make it big.
Committing an insert of millions of rows will take some time. Depending on your requirements you might want to commit the insert transaction every once in a while (every 250.000 inserts?)
Think about your transaction isolation: Make sure auto-commit is turned off as committing each insert will make most of the batching gains go away.
Related
This is not the rare question on the net, but I does a few optimization work with MySQL server for solve this problem and did not get results. So at first I use maven's package mysql:mysql-connector-java:6.0.6.
I try just to run this code:
try {
mysqlConnection = DriverManager.getConnection(DatabaseUtils.mysqlUrl, DatabaseUtils.mysqlUser, DatabaseUtils.mysqlPassword);
PreparedStatement valuesStatement = "SELECT * FROM `test` ORDER BY `id`"
ResultSet cursor = valuesStatement.executeQuery();
double value = 0;
if (cursor.next())
value = cursor.getDouble("value");
} catch (SQLException sqlEx) {
sqlEx.printStackTrace();
} finally {
cursor.close();
pricesStatement.close();
}
I have a lot records in the table. It's about million but every day add about thousand records. So I was very surprised when this simple example executed 30 seconds. I googled my problem and I find only "using pool", "tune mysql server", "try to EXPLAIN SELECT". But I've noticed that execution time related with rows count. So I looked into driver's code and found that:
TextResultsetReader::read():
while(true) {
if(row == null) {
rows = new ResultsetRowsStatic(rowList, cdef);
break;
}
if(maxRows == -1 || rowList.size() < maxRows) {
rowList.add(row);
}
row = (ResultsetRow)this.protocol.read(ResultsetRow.class, trf);
}
This means that even if I want to fetch only one row driver fetches all queried rows and get me first of it. Manuals suggest to use "setFetchSize" for fetching only n records. But it doesn't work. Driver code fetching all data anyway. So then I found that there is two recordsets: ResultRowsStatic and ResultSetStreaming. Second seems to be fetching data only when I need query it. How to use ResultRowsStreaming? I found it only into code. Parameter "fetchSize" must to equal -2147483648. I did try and it worked! Execution time of "executeQuery()" now if about 0.0007 sec. It's very fast for me. But wait.. My script anyway takes 30 seconds. Why? I debugged code after executing query. There's only two "close" methods after that. What's can go wrong? And that's true, "cursor.close()" takes the rest of time. I looked into library code again and reached ResultsetRowsStreaming::close():
boolean hadMore = false;
int howMuchMore = 0;
synchronized(mutex) {
while(this.next() != null) {
hadMore = true;
++howMuchMore;
if(howMuchMore % 100 == 0) {
Thread.yield();
}
}
if(conn != null) {
if(!((Boolean)this.protocol.getPropertySet().getBooleanReadableProperty("clobberStreamingResults").getValue()).booleanValue() && ((Integer)this.protocol.getPropertySet().getIntegerReadableProperty("netTimeoutForStreamingResults").getValue()).intValue() > 0) {
int oldValue = this.protocol.getServerSession().getServerVariable("net_write_timeout", 60);
this.protocol.clearInputStream();
try {
this.protocol.sqlQueryDirect((StatementImpl)null, "SET net_write_timeout=" + oldValue, (String)this.protocol.getPropertySet().getStringReadableProperty("characterEncoding").getValue(), (PacketPayload)null, -1, false, (String)null, (ColumnDefinition)null, (GetProfilerEventHandlerInstanceFunction)null, this.resultSetFactory);
} catch (Exception var9) {
throw ExceptionFactory.createException(var9.getMessage(), var9, this.exceptionInterceptor);
}
}
if(((Boolean)this.protocol.getPropertySet().getBooleanReadableProperty("useUsageAdvisor").getValue()).booleanValue() && hadMore) {
ProfilerEventHandler eventSink = ProfilerEventHandlerFactory.getInstance(conn.getSession());
eventSink.consumeEvent(new ProfilerEventImpl(0, "", this.owner.getCurrentCatalog(), this.owner.getConnectionId(), this.owner.getOwningStatementId(), -1, System.currentTimeMillis(), 0L, Constants.MILLIS_I18N, (String)null, (String)null, Messages.getString("RowDataDynamic.2") + howMuchMore + Messages.getString("RowDataDynamic.3") + Messages.getString("RowDataDynamic.4") + Messages.getString("RowDataDynamic.5") + Messages.getString("RowDataDynamic.6") + this.owner.getPointOfOrigin()));
}
}
}
This code unconditionally fetching all the rest of data only for logging how many records I did not fetched. Really weird. And it would be justified if logger was attached. But in my case this code counting unfetched rows and in 30 seconds and... do nothing with it. And this proble I cannot fix because there's not parameter which can tell code not to count rows.
Now I don't know what to do next. Query time is very slow for me. For example mysql driver for php execute this query in 0.0004-0.001 seconds.
So people who using mysql-connector for Java, tell me please have you got these problems? If not, could you post any examples what should I do to bypass the above problems? Maybe you use another connectors. So tell me please, what to do?
Your SQL query says
SELECT * FROM test ORDER BY id
You are, with that query, instructing your MySQL server to serialize every column of every row of your test table and send it to your Java program. So, MySQL obeys. You have a large table. So your instruction to MySQL takes time. And yes, the more rows in your table the longer it takes. This is not a problem with JDBC or the driver; it's a problem with the SQL you're using.
It seems from your sample code that you want one column -- named value -- from one row -- the first one -- in your table. You could accomplish that using this SQL statement:
SELECT value FROM test ORDER BY id LIMIT 1
If your id column is your table's primary key, this will be fast.
The whole point of SQL is to allow your tables to contain so many rows that it's unreasonable to fetch them all into your Java (or other) program in a short amount of time. That's why SQL has WHERE and LIMIT clauses.
Below is a sample of what I did
{
...
Class.forName("com.mysql.jdbc.Driver").newInstance();
con=DriverManager.getConnection(ConnectionStr,"root","root");
String prepareStr="DELETE FROM customer_maintenance where id=?";
PreparedStatement pst= con.prepareStatement(prepareStr);
pst.setInt(1,id);
pst.executeUpdate();
String update_key="SET #count = 0; UPDATE customer_maintenance SET customer_maintenance.id = #count:= #count + 1;";
PreparedStatement pst1=con.prepareStatement(update_key);
pst1.executeUpdate();
System.out.println("UPDATE");
}
It is throwing the following exception
com.mysql.jdbc.exceptions.MySQLSyntaxErrorException:
You have an error in your SQL syntax
and I am not sure if it is possible to update the primary key, every time a row is deleted.
You should provide the exception you get, but the first problem I see in your code is that you may try to update a row setting ID = x, while another row could still have ID = x, thus an error for duplicate key.
That's because that update statement won't guarantee that the rows are updated following the ID order.
With the values in your example:
Start Delete Update1 Update2
ID ID ID ID
1 (del)
2 2 2 2
3 3 1 (upd) 1
4 4 4 2 (upd) <- duplicate key
Edit
Easiest fix is adding an order by clause to your query, so that the rows are update starting from the smaller ID.
You should also get rid of the ; at the end of the query.
String update_key="SET #count = 0;
UPDATE customer_maintenance
SET customer_maintenance.id = #count:= #count + 1
ORDER BY customer_maintenance.id";
If you have set auto increment and primary key for Id column while table creation. Auto increment will handle this automatically while database deletion, no need to worry about it.
I have a Java program reading data from an Access database where the table is created dynamically each time, and the number of columns varies depending on the data populated.
The table has columns as shown below. Only columns RowID and StatusOfProcessing are fixed and will be at the end.
column1,column2,column3, ... columnN,RowID,StatusOfProcessing
Below is piece of code
String str = "SELECT TOP 50 * FROM Dynamic_table WHERE StatusOfProcessing='0'";
resultSet = stmt.executeQuery(str);
When reading data from the ResultSet, does it always have columns in the order listed above, or should I use
String str = "SELECT TOP 50 column1,column2,column3 .... columnN,RowID,StatusOfProcessing FROM Dynamic_table WHERE StatusOfProcessing='0'";
resultSet = stmt.executeQuery(str);
Can someone clarify?
SELECT * will normaly return columns in the order in which they were created, e.g., the order in which they appeared in the CREATE TABLE statement. However, bear in mind that in addition to retrieving the value of a column by its index, as in
int col1value = resultSet.getInt(1);
you can also retrieve the value in a given column by referring to its name, as in
int col1value = resultSet.getInt("column1");
Furthermore, if you need to see the actual order of the columns or verify the names and types of the columns in a ResultSet you can use a ResultSetMetaData object to get that information.
I'm having a problem with a java OutOfMemoryError. The program basically looks at mysql tables that are running on mysql workbench, and queries them to get out certain information, and then puts them in CSV files.
The program works just fine with a smaller data set, but once I use a larger data set (hours of logging information as opposed to perhaps 40 minutes) I get this error, which to me says that the problem comes from having a huge data set and the information not being handled too well by the program. Or it not being possible to handle this amount of data in the way that I have.
Setting Java VM arguments to -xmx1024m worked for a slightly larger data set but i need it to handle even bigger ones but it gives the error.
Here is the method which I am quite sure is the cause of the program somewhere:
// CSV is csvwriter (external lib), sment are Statements, rs is a ResultSet
public void pidsforlog() throws IOException
{
String[] procs;
int count = 0;
String temp = "";
System.out.println("Commence getting PID's out of Log");
try {
sment = con.createStatement();
sment2 = con.createStatement();
String query1a = "SELECT * FROM log, cpuinfo, memoryinfo";
rs = sment.executeQuery(query1a);
procs = new String[countThrough(rs)];
// SIMPLY GETS UNIQUE PROCESSES OUT OF TABLES AND STORES IN ARRAY
while (rs.next()) {
temp = rs.getString("Process");
if(Arrays.asList(procs).contains(temp)) {
} else {
procs[count] = temp;
count++;
}
}
// BELIEVE THE PROBLEM LIES BELOW HERE. SIZE OF THE RESULTSET TOO BIG?
for(int i = 0; i < procs.length; i++) {
if(procs[i] == null) {
} else {
String query = "SELECT DISTINCT * FROM log, cpuinfo, memoryinfo WHERE log.Process = " + "'" + procs[i] + "'" + " AND cpuinfo.Process = " + "'" + procs[i] + "'" + " AND memoryinfo.Process = " + "'" + procs[i] + "' AND log.Timestamp = cpuinfo.Timestamp = memoryinfo.Timestamp";
System.out.println(query);
rs = sment.executeQuery(query);
writer = new CSVWriter(new FileWriter(procs[i] + ".csv"), ',');
writer.writeAll(rs, true);
writer.flush();
}
}
writer.close();
} catch (SQLException e) {
notify("Error pidslog", e);
}
}; // end of method
Please feel free to ask if you want source code or more information as I'm desperate to get this fixed!
Thanks.
SELECT * FROM log, cpuinfo, memoryinfo will sure give a huge result set. It will give a cartesian product of all rows in all 3 tables.
Without seeing the table structure (or knowing the desired result) it's hard to pinpoint a solution, but I suspect that you either want some kind of join conditions to limit the result set, or use a UNION a'la;
SELECT Process FROM log
UNION
SELECT Process FROM cpuinfo
UNION
SELECT Process FROM memoryinfo
...which will just give you all distinct values for Process in all 3 tables.
Your second SQL statement also looks a bit strange;
SELECT DISTINCT *
FROM log, cpuinfo, memoryinfo
WHERE log.Process = #param1
AND cpuinfo.Process = #param1
AND memoryinfo.Process = #param1
AND log.Timestamp = cpuinfo.Timestamp = memoryinfo.Timestamp
Looks like you're trying to select from all 3 logs simultaneously, but ending up with another cartesian product. Are you sure you're getting the result set you're expecting?
You could limit the result returned by your SQL queryes with the LIMIT estatementet.
For example:
SELECT * FROM `your_table` LIMIT 100
This will return the first 100 results
SELECT * FROM `your_table` LIMIT 100, 200
This will return results from 100 to 200
Obviously you can iterate with those values so you get to all the elements on the data base no matter how many there are.
I think your are loading too many data at the same in the memory. try to use offset and limit in your sql statement so that you can avoid this problem
Your Java code is doing things that the database could do more efficiently. From query1a, it looks like all you really want is the unique processes. select distinct Process from ... should be sufficient to do that.
Then, think carefully about what table or tables are needed in that query. Do you really need log, cpuinfo, and memoryinfo? As Joachim Isaksson mentioned, this is going to return the Cartesian product of those three tables, giving you x * y * z rows (where x, y, and z are the row counts in each of those three tables) and a + b + c columns (where a, b, and c are the column counts in each of the tables). I doubt that's what you want or need. I assume you could get those unique processes from one table, or a union (rather than join) of the three tables.
Lastly, your second loop and query are essentially doing a join, something again better and more efficiently left to the database.
Like others said, fetching the data in smaller chunks might resolve the issue.
This is one of the other threads in stackoverflow that talks about this issue:
How to read all rows from huge table?
I have a SELECT query which combining three tables. I want to add them to a Jtable by separating the MYSQL tables. I just want to know how can I identify the table name in a Resultset?
resultLoad("SELECT sqNum,RegDate,JobNo,DecName,NoOfLines,EntryType,EntrySubType,EntrySubType2,Amount,Status FROM paysheet WHERE TypeBy='" + cmbStaff.getSelectedItem().toString() + "' AND CommissionStatus='no' UNION SELECT sqNum,RegDate,JobNo,DecName,NoOfLines,EntryType,EntrySubType,EntrySubType2,Amount,Status FROM creditsheet WHERE TypeBy='" + cmbStaff.getSelectedItem().toString() + "' AND CommissionStatus='no' ORDER BY RegDate UNION SELECT sqNumber,date,JobNumber,DecName,noOfLines,type,type,type,CommissionAmount,status FROM newjobsheet WHERE TypeBy='" + cmbStaff.getSelectedItem().toString() + "' AND CommissionStatus='no' ORDER BY RegDate");
private void resultLoad(String loadQuery) {
try {
Connection c = DB.myConnection();
Statement s = c.createStatement();
ResultSet r = s.executeQuery(loadQuery);
while (r.next()) {
Vector v = new Vector();
v.addElement(r.getString("sqNum"));
v.addElement(r.getString("RegDate"));
v.addElement(r.getString("JobNo"));
v.addElement(r.getString("DecName"));
v.addElement(r.getString("NoOfLines"));
v.addElement(r.getString("EntryType") + " - " + r.getString("EntrySubType") + " - " + r.getString("EntrySubType2"));
v.addElement(r.getString("Amount"));
v.addElement(r.getString("Status"));
tm.addRow(v);
totalComAmount = totalComAmount + Integer.parseInt(r.getString("Amount"));
}
} catch (Exception e) {
// e.printStackTrace();
JOptionPane.showMessageDialog(this, e, "Error!", JOptionPane.ERROR_MESSAGE);
}
}
I want to add to the Jtable like this by sorting the dates. But the three tables containing different columns.
From your result set, you can get ResultSetMetaData. It looks like this:
rs.getMetaData().getTableName(int Column);
"I want to add them to a table by separating the tables."
Not sure what you mean by that, but:
"I just want to know how can I identify the table name in a Resultset?"
the answer is no, not unless you rewrite the query so that it does it for you.
A SELECT statement yields a new (virtual) table - any columns it delivers are technically columns of that new virtual table. The result does not remember the origin of those columns
However, you can write the query so as to give every expression in the SELECT list a column alias that allows you to identify the origin. For instance, you could do:
SELECT table1.column1 AS table1_column1
, table1.column2 AS table1_column2
, ...
, table2.column1 AS table2_column1
, ...
FROM table1 INNER JOIN table2 ON ... etc
If the underscore is not suitable as a separator for your purpose, then you could consider to quote the aliases and pick whatever character as separator you like, including a dot:
SELECT table1.column1 AS `table1.column1`
, ....
etc.
UPDATE:
I just noticed your query is a UNION over several tables. Obviously this method won't work in that case. I think your best bet is still to rewrite the query so that it reads:
SELECT 'paysheet' as table_name, ...other columns...
FROM paysheet
UNION
SELECT 'creditsheet', ...other columns...
...
It seems to me like you want to SELECT data from three separate tables and have it returned in one query as one ResultSet, then you want to separate the results back out into individual tables.
This is completely counter intuitive.
If you want to have your results separated so that you are able to look at each table's results separately, your best bet is to perform 3 different SELECT queries. Basically, what you are asking it to combine 3 things and then separate them again in one pass. Don't do this; if you leave them uncombined in the first place, separating them back out won't be an issue for you.