We Keep Coding

Insert statement contains more than 1000 elements

I'm trying to add 100,000 names to a database using JDBC. I'm aware that MS SQL doesnt allow mass inserts of more than 1000, elmts, so I accomplish this by breaking the major set down into sets containing 1000 or fewer. The following is my code:
StringJoiner joiner = new StringJoiner("\'), (\'", "INSERT INTO Names (Name) VALUES (\'", "\');");
ExecutorService threadpool = Executors.newFixedThreadPool(100);
while(names.size() > 0) {
int count = Math.min(1000, names.size());
HashSet<String> set = new HashSet();
Iterator iterator = names.iterator();
for(int i = 0; i < count; i++) {
set.add((String) iterator.next());
}
names.removeAll(set);
for(String s: set)
{
joiner.add(s);
}
// System.out.println(joiner.toString());
threadpool.submit(() -> {
PreparedStatement query = null;
try {
query = connect.prepareStatement(joiner.toString());
} catch (SQLException e) {
e.printStackTrace();
}
try {
query.executeUpdate();
} catch (SQLException e) {
e.printStackTrace();
}
});
}
}
This throws the following exception: com.microsoft.sqlserver.jdbc.SQLServerException: The number of row value expressions in the INSERT statement exceeds the maximum allowed number of 1000 row values.
Why are there more than 1000 lines being added?

Why not use batch inserts? Define batch size 1000 and insert queries in the batches of 1000 queries at a time.
String sql = "insert into names(name) values (?)";
Connection connection = new getConnection();
PreparedStatement ps = connection.prepareStatement(sql);
final int batchSize = 1000;
int count = 0;
for (Employee employee: employees) {
ps.setString(1, employee.getName());
ps.addBatch();
if(++count % batchSize == 0) {
ps.executeBatch();
}
}
ps.executeBatch(); // insert remaining records
ps.close();
connection.close();

Storing Multiple Rows from MySQL query in MultiDimensional HashMap

So I'm trying to store a MySQL query result set into a multi dimensional HashMap as listed so:
public HashMap<String, HashMap<String, String>> getData(String query)
{
Statement stmt = null;
HashMap<String, HashMap<String, String>> results = new HashMap<String, HashMap<String, String>>();
try
{
stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
ResultSetMetaData rsmd = rs.getMetaData();
while (rs.next())
{
for (int i = 1; i < rsmd.getColumnCount() + 1; i++)
{
results.put(Integer.toString(i - 1), new HashMap<String, String>());
results.get(Integer.toString(i - 1)).put(rsmd.getColumnLabel(i), rs.getString(i));
}
}
}
catch (SQLException ex)
{
ex.printStackTrace(System.out);
}
return results;
}
However when using the function to print it out as so:
public static void printMap(Map mp)
{
Iterator it = mp.entrySet().iterator();
while (it.hasNext())
{
Map.Entry pair = (Map.Entry)it.next();
System.out.println(pair.getKey() + " = " + pair.getValue());
it.remove();
}
}
It is only storing a single row result and I can't wrap my head around why.
0 = {Date=2014-11-04}
1 = {Num=1256}
2 = {ATime=null}
3 = {ALocCode=null}
4 = {DTime=1:00 PM}
5 = {DLocCode=JFK}
6 = {EstATime=8:00 PM}
7 = {EstDTime=1:00 PM}
8 = {EId=7624}
My question is, and the only way I can put it is relating to PHP, is how can I make it store like this?
$result[0]['Date'] = '3214';
....
$result[1]['Date'] = '6426';
Since that is essentially what I'm trying to achieve?

main problem that you've swapped "rows" and "columns", next one is that you're re-creating HashMap every time you put field, proper code will look like this:
public Map<String, Map<String, String>> getData(final String query) {
final Map<String, Map<String, String>> results = new HashMap<>();
try (final Statement stmt = this.conn.createStatement(); final ResultSet rs = stmt.executeQuery(query);) {
final ResultSetMetaData rsmd = rs.getMetaData();
long rId = 0;
while (rs.next()) {
final Map<String, String> record = new HashMap<>();
for (int i = 1; i < (rsmd.getColumnCount() + 1); i++) {
record.put(rsmd.getColumnLabel(i), rs.getString(i));
}
results.put(String.valueOf(rId++), record);
}
} catch (final SQLException ex) {
ex.printStackTrace(System.out);
}
return results;
}
public static void printMap(final Map<?, ?> mp) {
for (final Entry<?, ?> entry : mp.entrySet()) {
final Object key = entry.getKey();
final Object value = entry.getValue();
if (value instanceof Map) {
System.out.println(key);
printMap((Map<?, ?>) value);
} else {
System.out.println(key + "=" + entry.getValue());
}
}
}

The answer by Lashane is good for the errors you needed solving, however it can be improved:
You wanted numeric access ($result[0]['Date']) to the rows, not string.
print method should use fully typed parameter.
Rows should be stored in TreeMap or LinkedHashMap or ArrayList to retain row order. ArrayList is better for your case, actually.
Columns should be stored in LinkedHashMap to retain column order.
Do not catch exception and continue. Allow it to cascade up to caller.
Updated version:
public List<Map<String, String>> getData(final String query) throws SQLException {
final List<Map<String, String>> results = new ArrayList<>();
try (Statement stmt = this.conn.createStatement();
ResultSet rs = stmt.executeQuery(query)) {
ResultSetMetaData metaData = rs.getMetaData();
while (rs.next()) {
Map<String, String> record = new LinkedHashMap<>();
for (int col = 1; col <= metaData.getColumnCount(); col++)
record.put(metaData.getColumnLabel(col), rs.getString(col));
results.add(record);
}
}
return results;
}
public static void printMap(List<Map<String, String>> rows) {
for (int rowNum = 0; rowNum < rows.size(); rowNum++)
System.out.println(rowNum + " = " + rows.get(rowNum));
}
You can now access it like you did in PHP:
// PHP (for reference, the way you requested)
$result[0]['Date']
// Java
result.get(0).get("Date")
// Groovy
result[0]['Date']
result[0].Date
// JSP
<c:forEach var="row" items="${result}" varStatus="rowStatus">
${rowStatus.index} = <c:out value="${row.Date}"/>, ...
</c:forEach>

Java HashMap inside ArrayList

How can I get the value of HashTable inside the arrayList?
I have the following code:
public ArrayList resultSetToArrayList(ResultSet rs) {
ArrayList list = new ArrayList();
try {
ResultSetMetaData md = rs.getMetaData();
int columns = md.getColumnCount();
int rowcount = 0;
while (rs.next()) {
Hashtable row = new Hashtable();
for (int i = 1; i <= columns; ++i) {
row.put(md.getColumnName(i), rs.getString(i));
}
list.add(rowcount, row);
rowcount++;
}
} catch (Exception e) {
e.printStackTrace();
}
return list;
}

You appear to be using raw types and Hashtable instead of a HashMap. I think you're asking for something like
public List<Map<String, String>> resultSetToArrayList(ResultSet rs) {
List<Map<String, String>> list = new ArrayList<>();
try {
ResultSetMetaData md = rs.getMetaData();
int columns = md.getColumnCount();
while (rs.next()) {
Map<String, String> row = new HashMap<>();
for (int i = 1; i <= columns; ++i) {
row.put(md.getColumnName(i), rs.getString(i));
}
list.add(row);
}
} catch (Exception e) {
e.printStackTrace();
}
return list;
}
As for getting the values back out of a Map, you might iterate the Map.keySet() like
for (String key : map.keySet()) {
System.out.printf("%s = %s%n", key, map.get(key));
}

Efficient way to Handle ResultSet in Java

I'm using a ResultSet in Java, and am not sure how to properly close it. I'm considering using the ResultSet to construct a HashMap and then closing the ResultSet after that. Is this HashMap technique efficient, or are there more efficient ways of handling this situation? I need both keys and values, so using a HashMap seemed like a logical choice.
If using a HashMap is the most efficient method, how do I construct and use the HashMap in my code?
Here's what I've tried:
public HashMap resultSetToHashMap(ResultSet rs) throws SQLException {
ResultSetMetaData md = rs.getMetaData();
int columns = md.getColumnCount();
HashMap row = new HashMap();
while (rs.next()) {
for (int i = 1; i <= columns; i++) {
row.put(md.getColumnName(i), rs.getObject(i));
}
}
return row;
}

Iterate over the ResultSet
Create a new Object for each row, to store the fields you need
Add this new object to ArrayList or Hashmap or whatever you fancy
Close the ResultSet, Statement and the DB connection
Done
EDIT: now that you have posted code, I have made a few changes to it.
public List resultSetToArrayList(ResultSet rs) throws SQLException{
ResultSetMetaData md = rs.getMetaData();
int columns = md.getColumnCount();
ArrayList list = new ArrayList(50);
while (rs.next()){
HashMap row = new HashMap(columns);
for(int i=1; i<=columns; ++i){
row.put(md.getColumnName(i),rs.getObject(i));
}
list.add(row);
}
return list;
}

I just cleaned up RHT's answer to eliminate some warnings and thought I would share. Eclipse did most of the work:
public List<HashMap<String,Object>> convertResultSetToList(ResultSet rs) throws SQLException {
ResultSetMetaData md = rs.getMetaData();
int columns = md.getColumnCount();
List<HashMap<String,Object>> list = new ArrayList<HashMap<String,Object>>();
while (rs.next()) {
HashMap<String,Object> row = new HashMap<String, Object>(columns);
for(int i=1; i<=columns; ++i) {
row.put(md.getColumnName(i),rs.getObject(i));
}
list.add(row);
}
return list;
}

RHT pretty much has it. Or you could use a RowSetDynaClass and let someone else do all the work :)

this is my alternative solution, instead of a List of Map, i'm using a Map of List.
Tested on tables of 5000 elements, on a remote db, times are around 350ms for eiter method.
private Map<String, List<Object>> resultSetToArrayList(ResultSet rs) throws SQLException {
ResultSetMetaData md = rs.getMetaData();
int columns = md.getColumnCount();
Map<String, List<Object>> map = new HashMap<>(columns);
for (int i = 1; i <= columns; ++i) {
map.put(md.getColumnName(i), new ArrayList<>());
}
while (rs.next()) {
for (int i = 1; i <= columns; ++i) {
map.get(md.getColumnName(i)).add(rs.getObject(i));
}
}
return map;
}

A couple of things to enhance the other answers. First, you should never return a HashMap, which is a specific implementation. Return instead a plain old java.util.Map. But that's actually not right for this example, anyway. Your code only returns the last row of the ResultSet as a (Hash)Map. You instead want to return a List<Map<String,Object>>. Think about how you should modify your code to do that. (Or you could take Dave Newton's suggestion).

i improved the solutions of RHTs/Brad Ms and of Lestos answer.
i extended both solutions in leaving the state there, where it was found.
So i save the current ResultSet position and restore it after i created the maps.
The rs is the ResultSet, its a field variable and so in my solutions-snippets not visible.
I replaced the specific Map in Brad Ms solution to the gerneric Map.
public List<Map<String, Object>> resultAsListMap() throws SQLException
{
var md = rs.getMetaData();
var columns = md.getColumnCount();
var list = new ArrayList<Map<String, Object>>();
var currRowIndex = rs.getRow();
rs.beforeFirst();
while (rs.next())
{
HashMap<String, Object> row = new HashMap<String, Object>(columns);
for (int i = 1; i <= columns; ++i)
{
row.put(md.getColumnName(i), rs.getObject(i));
}
list.add(row);
}
rs.absolute(currRowIndex);
return list;
}
In Lestos solution, i optimized the code. In his code he have to lookup the Maps each iteration of that for-loop. I reduced that to only one array-acces each for-loop iteration. So the program must not seach each iteration step for that string-key.
public Map<String, List<Object>> resultAsMapList() throws SQLException
{
var md = rs.getMetaData();
var columns = md.getColumnCount();
var tmp = new ArrayList[columns];
var map = new HashMap<String, List<Object>>(columns);
var currRowIndex = rs.getRow();
rs.beforeFirst();
for (int i = 1; i <= columns; ++i)
{
tmp[i - 1] = new ArrayList<>();
map.put(md.getColumnName(i), tmp[i - 1]);
}
while (rs.next())
{
for (int i = 1; i <= columns; ++i)
{
tmp[i - 1].add(rs.getObject(i));
}
}
rs.absolute(currRowIndex);
return map;
}

Here is the code little modified that i got it from google -
List data_table = new ArrayList<>();
Class.forName("oracle.jdbc.driver.OracleDriver");
con = DriverManager.getConnection(conn_url, user_id, password);
Statement stmt = con.createStatement();
System.out.println("query_string: "+query_string);
ResultSet rs = stmt.executeQuery(query_string);
ResultSetMetaData rsmd = rs.getMetaData();
int row_count = 0;
while (rs.next()) {
HashMap<String, String> data_map = new HashMap<>();
if (row_count == 240001) {
break;
}
for (int i = 1; i <= rsmd.getColumnCount(); i++) {
data_map.put(rsmd.getColumnName(i), rs.getString(i));
}
data_table.add(data_map);
row_count = row_count + 1;
}
rs.close();
stmt.close();
con.close();

public static List<HashMap<Object, Object>> GetListOfDataFromResultSet(ResultSet rs) throws SQLException {
ResultSetMetaData metaData = rs.getMetaData();
int count = metaData.getColumnCount();
String[] columnName = new String[count];
List<HashMap<Object,Object>> lst=new ArrayList<>();
while(rs.next()) {
HashMap<Object,Object> map=new HashMap<>();
for (int i = 1; i <= count; i++){
columnName[i-1] = metaData.getColumnLabel(i);
map.put(columnName[i-1], rs.getObject(i));
}
lst.add(map);
}
return lst;
}

How to improve performance while update column Clob in oracle?

I used the following codes to update column Clob in oracle, it seems to be okay and work properly, after performance testing, it reported that need consumed more than 200ms while the length of string is more than 130000. Is it any good way to improve it?
private void updateClobDetailsField(Map<Integer, String> idToDetails){
long s1 = System.currentTimeMillis();
Connection conn = null;
PreparedStatement pStmt = null;
ResultSet rset = null;
Map<Integer, Clob> idToDetailsClob = new HashMap<Integer, Clob>();
int BATCH_SIZE = CMType.BATCH_UPDATE_MAXSIZE;
try
{
conn = getConnection();
ServerAdapter adapter = ServerAdapter.getServerAdapter();
List<Integer> IDList = new ArrayList<Integer>();
for(Integer id : idToDetails.keySet()){
IDList.add(id);
}
List<Integer> tempIDList = new ArrayList<Integer>(IDList);
while(!tempIDList.isEmpty()){
int size = tempIDList.size() < BATCH_SIZE ? tempIDList.size() : BATCH_SIZE;
List<Integer> currentBatch = tempIDList.subList(0, size);
String inClause = SQLHelper.prepareInClause("ID",currentBatch.size());
pStmt = conn.prepareStatement("SELECT ID, DETAILS FROM PROGRAM_HISTORY WHERE " + inClause);
for(int i = 0; i < currentBatch.size(); i++){
pStmt.setInt(i+1, (currentBatch.get(i)));
}
rset = pStmt.executeQuery();
while(rset.next()){
int id = rset.getInt(1);
Clob detailsClob = rset.getClob(2);
Writer writer = adapter.getCharacterOutputStream(detailsClob);
String details = idToDetails.get(id);
if (details != null) {
writer.write(details);
}
writer.flush();
writer.close();
idToDetailsClob.put(id, detailsClob);
}
currentBatch.clear();
BaseSQLHelper.close(pStmt, rset);
}
int counter = 0;
pStmt = conn.prepareStatement("UPDATE PROGRAM_HISTORY SET DETAILS = ? WHERE ID = ?");
for(int i=0; i<IDList.size(); i++){
int index = 1;
Clob detailsClob = (Clob) idToDetailsClob.get(IDList.get(i));
pStmt.setClob(index++, detailsClob);
pStmt.setInt(index++, IDList.get(i));
pStmt.addBatch();
counter++;
if(counter % BATCH_SIZE == 0) {
pStmt.executeBatch();
pStmt.clearBatch();
counter = 0;
}
}
if(IDList.size() % BATCH_SIZE > 0) {
pStmt.executeBatch();
}
}
catch (SQLException se)
{
se.printStackTrace();
}
catch (IOException se)
{
se.printStackTrace();
}
finally
{
cleanup(conn, pStmt, null);
}
System.out.println(System.currentTimeMillis()-s1);
}

If I understand your code correctly, you are appending text to your details clob column.
Doing it in PL/SQL would be faster since you wouldn't have to fetch the clob across the network. For example you could prepare this statement:
DECLARE
l_details CLOB;
BEGIN
SELECT details INTO l_details FROM program_history WHERE ID = ?;
dbms_lob.append(l_details, ?);
END;
and bind currentBatch.get(i) and idToDetails.get(id).
Notice that you don't need an additional update with PL/SQL.

Execute your query with an updatable ResultSet so that you can update the data as you scroll through without separate update statements being executed.
You need to create your prepared statement with the resultSetConcurrency set to ResultSet.CONCUR_UPDATABLE. Check out the oracle documentation on dealing with streams for the various ways you can handle the clob data.