I've a spring project where I need to export a query resultset into excel sheet. Currently I'm using JPA repository to get the data from the DB and i'm using Apache POI libraries to prepare excel sheet from this data.
// Get data from DB using jpaRepository
Page<MyPOJO> data = myPOJOJpaRepository.findAll(specifications, pageRequest);
// Prepare Excel Sheet from the data object using POI libraries
Now, the problem is getting the data in the form of Java POJOs is taking too much time (almost 60 secs) and preparing excel sheet using the POI libraries is also taking almost 60 secs.
When I tried to export a csv file instead using the resultset (instead of java POJOs), it is finishing in under 10 secs.
ResultSet resultSet = statement.executeQuery("select * from table where some_filters");
File file = writeResultsToCSVFile(resultSet);
I'm using JPA specifications to build the query in the current architecture. Is there anyway to get the query that is going to execute so that I can directly get the resultset (instead of POJOs) and prepare the csv file instead.
// I'm looking for something like follows:
ResultSet resultSet = statement.executeQuery(specifications.getQuery());
File file = writeResultsToCSVFile(resultSet);
Is there anyway to achieve something like this?
It's a bit tricky because you can obtain a non-standard query like this:
select generatedAlias0 from Pets as generatedAlias0 where generatedAlias0.pet_name=:param0
You have to obtain the query, then you need to manipulate something like requested fields and bound params (managing their types. note that in this example I managed only string type).
So assuming you are using Hibernate you can do something like this:
/**
*
*/
public static Specification<Pets> findByCriteria() {
return new Specification<Pets>() {
#Override
public Predicate toPredicate(Root<Pets> root, CriteriaQuery<?> query, CriteriaBuilder cb) {
List<Predicate> predicates = new ArrayList<Predicate>();
// solo attivita attive
predicates.add(cb.equal(root.get("pet_name"), "Chelsea"));
return cb.and(predicates.toArray(new Predicate[]{}));
}
};
}
/**
* TODO MANAGE VARIOUS TYPES
*/
private String createParam(Parameter<?> p, Query<?> q) {
Class<?> clz = p.getParameterType();
if (clz == String.class) {
return "'" + q.getParameterValue(p.getName()) + "'";
}
return "";
}
/**
*
*/
public void getEnterprisesAdmin() {
Specification<Pets> spec = this.findByCriteria();
CriteriaBuilder builder = this.em.getCriteriaBuilder();
CriteriaQuery<Pets> query = builder.createQuery(Pets.class);
Root<Pets> root = query.from(Pets.class);
Predicate predicate = spec.toPredicate(root, query, builder);
query.where(predicate);
TypedQuery<Pets> findAllBooks = em.createQuery(query);
Query<Pets> q = findAllBooks.unwrap(Query.class);
String strQuery = q.getQueryString();
strQuery = Pattern.compile("(.*?)select \\w*").matcher(strQuery).replaceFirst("SELECT *");
Set<Parameter<?>> pList = q.getParameters();
Iterator<Parameter<?>> iter = pList.iterator();
for (int i=0; i<pList.size(); i++) {
Parameter<?> p = iter.next();
strQuery = strQuery.replace(":" + p.getName(), this.createParam(p, q));
}
try {
Connection conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/mempoi?useSSL=false&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC", "root", "");
Statement stmt = conn.createStatement();
ResultSet resultSet = stmt.executeQuery(strQuery);
resultSet.next();
System.out.println("PET NAME: " + resultSet.getString("pet_name"));
} catch (Exception e) {
e.printStackTrace();
}
}
You have given me a good idea for the next feature to implement in my lib MemPOI (designed for managing cases like yours) that supplies an abstraction layer for Apache POI. I'll implement the export directly from a Specification
Related
This question already has answers here:
Using a variable instead of a parameter index with a JDBC prepared statement
(3 answers)
Closed 2 years ago.
Are there named parameters in JDBC instead of positional ones, like the #name, #city in the ADO.NET query below?
select * from customers where name=#name and city = #city
JDBC does not support named parameters. Unless you are bound to using plain JDBC (which causes pain, let me tell you that) I would suggest to use Springs Excellent JDBCTemplate which can be used without the whole IoC Container.
NamedParameterJDBCTemplate supports named parameters, you can use them like that:
NamedParameterJdbcTemplate jdbcTemplate = new NamedParameterJdbcTemplate(dataSource);
MapSqlParameterSource paramSource = new MapSqlParameterSource();
paramSource.addValue("name", name);
paramSource.addValue("city", city);
jdbcTemplate.queryForRowSet("SELECT * FROM customers WHERE name = :name AND city = :city", paramSource);
To avoid including a large framework, I think a simple homemade class can do the trick.
Example of class to handle named parameters:
public class NamedParamStatement {
public NamedParamStatement(Connection conn, String sql) throws SQLException {
int pos;
while((pos = sql.indexOf(":")) != -1) {
int end = sql.substring(pos).indexOf(" ");
if (end == -1)
end = sql.length();
else
end += pos;
fields.add(sql.substring(pos+1,end));
sql = sql.substring(0, pos) + "?" + sql.substring(end);
}
prepStmt = conn.prepareStatement(sql);
}
public PreparedStatement getPreparedStatement() {
return prepStmt;
}
public ResultSet executeQuery() throws SQLException {
return prepStmt.executeQuery();
}
public void close() throws SQLException {
prepStmt.close();
}
public void setInt(String name, int value) throws SQLException {
prepStmt.setInt(getIndex(name), value);
}
private int getIndex(String name) {
return fields.indexOf(name)+1;
}
private PreparedStatement prepStmt;
private List<String> fields = new ArrayList<String>();
}
Example of calling the class:
String sql;
sql = "SELECT id, Name, Age, TS FROM TestTable WHERE Age < :age OR id = :id";
NamedParamStatement stmt = new NamedParamStatement(conn, sql);
stmt.setInt("age", 35);
stmt.setInt("id", 2);
ResultSet rs = stmt.executeQuery();
Please note that the above simple example does not handle using named parameter twice. Nor does it handle using the : sign inside quotes.
Vanilla JDBC only supports named parameters in a CallableStatement (e.g. setString("name", name)), and even then, I suspect the underlying stored procedure implementation has to support it.
An example of how to use named parameters:
//uss Sybase ASE sysobjects table...adjust for your RDBMS
stmt = conn.prepareCall("create procedure p1 (#id int = null, #name varchar(255) = null) as begin "
+ "if #id is not null "
+ "select * from sysobjects where id = #id "
+ "else if #name is not null "
+ "select * from sysobjects where name = #name "
+ " end");
stmt.execute();
//call the proc using one of the 2 optional params
stmt = conn.prepareCall("{call p1 ?}");
stmt.setInt("#id", 10);
ResultSet rs = stmt.executeQuery();
while (rs.next())
{
System.out.println(rs.getString(1));
}
//use the other optional param
stmt = conn.prepareCall("{call p1 ?}");
stmt.setString("#name", "sysprocedures");
rs = stmt.executeQuery();
while (rs.next())
{
System.out.println(rs.getString(1));
}
You can't use named parameters in JDBC itself. You could try using Spring framework, as it has some extensions that allow the use of named parameters in queries.
Plain vanilla JDBC does not support named parameters.
If you are using DB2 then using DB2 classes directly:
Using named parameter markers with PreparedStatement objects
Using named parameter markers with CallableStatement objects
EDIT: New links for Db2 v11.5:
Using named parameter markers with PreparedStatement objects
Using named parameter markers with CallableStatement objects
How to run a .sql script (from file) in Java and return a ResultSet using Spring?
I have a program that runs SQL queries on a database that return ResultSet which I later process and use the data in my classes. I am currently using JDBC with the scripts inside the Java program.
StringBuilder query = new StringBuilder("some script on multiple lines");
PreparedStatement statement = connection.prepareStatement(query.toString());
ResultSet resultSet = statement.executeQuery();
I want to move the SQL queries outside of the Java program to .sql files, but I want to keep all the program logic from the executeQuery statements on. This means I want to have the queries return a ResultSet.
I looked to several methods like using a ScriptRunner, using Spring JdbcTestUtils.executeSqlScript or reading the .sql file using a BufferReader and then passing the string to my statement. The ScriptRunner and the Spring JdbcTestUtils.executeSqlScript seem to not return a ResultSet, or I couldn't find the proper implementation. I want to stay away of the BufferReader method since it will require text parsing and a lot of exceptions to handle.
ScriptRunner scriptRunner = new ScriptRunner(connection, true, true);
scriptRunner.runScript(new FileReader("script.sql"));
The runScript method returns void. Same does the Spring implementation:
MysqlDataSource ds = new MysqlDataSource();
ds.setServerName("host");
ds.setUser("user");
ds.setPassword("password");
JdbcTemplate jdbcTemplate = new JdbcTemplate(ds);
Resource resource = new ClassPathResource("script.sql");
JdbcTestUtils.executeSqlScript(jdbcTemplate, resource, true);
I looked thorough the Spring api but couldn't find something similar to what I want.
Is there a way to load a script from file and then run it so it returns a ResultSet using Spring? I would prefer using Spring since it is actively mantained.
I found a way to do it using Spring:
MysqlDataSource ds = new MysqlDataSource();
ds.setServerName("host");
ds.setUser("user");
ds.setPassword("password");
JdbcTemplate jdbcTemplate = new JdbcTemplate(ds);
BufferedReader in = new BufferedReader(new FileReader("script.sql"));
LineNumberReader fileReader = new LineNumberReader(in);
String query = JdbcTestUtils.readScript(fileReader);
Now we will use the jdbcTemplate.query to query the database using the .sql script we read. The ResultSet required will be passed as parameter to the extractData of ResultSetExtractor implementation or to the mapRow of the RowMapper implementation. So the processing of the ResultSet will be done in the extractData or mapRow implementation and we will return the Collection/Object we need
List<YourClass> result = jdbcTemplate.query(query, new RowMapper<YourClass>() {
#Override
public YourClass mapRow(ResultSet rs, int i) throws SQLException {
// processing of the ResultSet
return result;
}
});
or
YourClass object = jdbcTemplate.query(query, new ResultSetExtractor<YourClass>() {
#Override
public YourClass extractData(ResultSet rs) throws SQLException, DataAccessException {
// processing of the ResultSet
return result;
}
});
Of course, using the last implementation you can return any object you want, even a collection of objects.
I would use property file to store sql queries.
For example:
person-save=insert into person values....
person-get-all=select * from person....
With that you can read the statements from the file
In the context:
<util:properties id="sqlProps" location="classpath:sqlProps.properties" />
In the DAO:
#Autowired
#Qualifier("sqlProps")
private Properties sqlProps;
...
String query = sqlProps.getProperty("person-get-all");
PreparedStatement statement = connection.prepareStatement(query);
ResultSet resultSet = statement.executeQuery();
If you want to execute all the statements in a file, you have to implement that yourself. Just loop over the properties and execute them one by one. You will need some flag to determinate if its select or update/delete statement.
Loop over properties:
Enumeration e = props.propertyNames();
while (e.hasMoreElements()) {
String key = (String) e.nextElement();
String query= props.getProperty(key);
}
I am planning to use Datastax Java driver for writing to Cassandra.. I was mainly interested in Batch Writes and Asycnhronous features of Datastax java driver but I am not able to get any tutorials which can explain me how to incorporate these features in my below code which uses Datastax Java driver..
/**
* Performs an upsert of the specified attributes for the specified id.
*/
public void upsertAttributes(final String userId, final Map<String, String> attributes, final String columnFamily) {
try {
// make a sql here using the above input parameters.
String sql = sqlPart1.toString()+sqlPart2.toString();
DatastaxConnection.getInstance();
PreparedStatement prepStatement = DatastaxConnection.getSession().prepare(sql);
prepStatement.setConsistencyLevel(ConsistencyLevel.ONE);
BoundStatement query = prepStatement.bind(userId, attributes.values().toArray(new Object[attributes.size()]));
DatastaxConnection.getSession().execute(query);
} catch (InvalidQueryException e) {
LOG.error("Invalid Query Exception in DatastaxClient::upsertAttributes "+e);
} catch (Exception e) {
LOG.error("Exception in DatastaxClient::upsertAttributes "+e);
}
}
In the below code, I am creating a Connection to Cassandra nodes using Datastax Java driver.
/**
* Creating Cassandra connection using Datastax Java driver
*
*/
private DatastaxConnection() {
try{
builder = Cluster.builder();
builder.addContactPoint("some_nodes");
builder.poolingOptions().setCoreConnectionsPerHost(
HostDistance.LOCAL,
builder.poolingOptions().getMaxConnectionsPerHost(HostDistance.LOCAL));
cluster = builder
.withRetryPolicy(DowngradingConsistencyRetryPolicy.INSTANCE)
.withReconnectionPolicy(new ConstantReconnectionPolicy(100L))
.build();
StringBuilder s = new StringBuilder();
Set<Host> allHosts = cluster.getMetadata().getAllHosts();
for (Host h : allHosts) {
s.append("[");
s.append(h.getDatacenter());
s.append(h.getRack());
s.append(h.getAddress());
s.append("]");
}
System.out.println("Cassandra Cluster: " + s.toString());
session = cluster.connect("testdatastaxks");
} catch (NoHostAvailableException e) {
e.printStackTrace();
throw new RuntimeException(e);
} catch (Exception e) {
}
}
Can anybody help me on how to add Batch writes or Asynchronous features to my above code.. Thanks for the help..
I am running Cassandra 1.2.9
For asynch it's as simple as using the executeAsync function:
...
DatastaxConnection.getSession().executeAsync(query);
For the batch, you need to build the query (I use strings because the compiler knows how to optimize string concatenation really well):
String cql = "BEGIN BATCH "
cql += "INSERT INTO test.prepared (id, col_1) VALUES (?,?); ";
cql += "INSERT INTO test.prepared (id, col_1) VALUES (?,?); ";
cql += "APPLY BATCH; "
DatastaxConnection.getInstance();
PreparedStatement prepStatement = DatastaxConnection.getSession().prepare(cql);
prepStatement.setConsistencyLevel(ConsistencyLevel.ONE);
// this is where you need to be careful
// bind expects a comma separated list of values for all the params (?) above
// so for the above batch we need to supply 4 params:
BoundStatement query = prepStatement.bind(userId, "col1_val", userId_2, "col1_val_2");
DatastaxConnection.getSession().execute(query);
On a side note, I think your binding of the statement might look something like this, assuming you change attributes to a list of maps where each map represents an update/insert inside the batch:
BoundStatement query = prepStatement.bind(userId,
attributesList.get(0).values().toArray(new Object[attributes.size()]),
userId_2,
attributesList.get(1).values().toArray(new Object[attributes.size()]));
For the example provided in Lyuben's answer, setting certain attributes of a batch like Type.COUNTER (if you need to update counters) using strings won't work. Instead you can arrange your prepared statements in batch like so:
final String insertQuery = "INSERT INTO test.prepared (id, col_1) VALUES (?,?);";
final PreparedStatement prepared = session.prepare(insertQuery);
final BatchStatement batch = new BatchStatement(BatchStatement.Type.UNLOGGED);
batch.add(prepared.bind(userId1, "something"));
batch.add(prepared.bind(userId2, "another"));
batch.add(prepared.bind(userId3, "thing"));
session.executeAsync(batch);
Using PreparedStatement to build a query that looks like this...
SELECT * FROM table1 WHERE column1 IN ('foo', 'bar')
...without knowing the number of strings in the IN statement
Constructing a string like...
"'foo', 'bar'"
...and passing that in with ps.setString() results in:
"\'foo\', \'bar\'"
Which is probably a good thing, but it makes this approach to my problem useless.
Any ideas on how to pass in an unknown number of values into a JDBC PreparedStatement without dynamically creating the query string too (this query lives in a file for easy reuse and I'd like to keep it that way)?
I tend to use a method that will modify the query to modify the query accordingly. This is a basic example that omits error handling for simplicity:
public String addDynamicParameters(String query, List<Object> parameters) {
StringBuilder queryBuilder = new StringBuilder(query);
queryBuilder.append("?");
for (int i = 1; i < parameters.size(); i++) {
queryBuilder.append(", ?");
}
queryBuilder.append(") ");
return queryBuilder.toString();
}
public void addParameters(PreparedStatement pstmt, List<Object> parameters) {
int i = 1;
for(Object param : parameters) {
pstmt.setObject(i++, param);
}
}
public void testDynamicParameters() {
String query = "SELECT col3 FROM tableX WHERE col1 = ? AND col2 IN (";
List<Object> parametersForIn = ...;
query = addDynamicParameters(query, parametersForIn);
List<Object> parameters = ...;
PreparedStatement pstmt = ...; //using your Connection object...
parameters.addAll(parametersForIn);
addParameters(pstmt, parameters);
//execute prepared statement...
//clean resources...
}
I'm writing data from Java to an Access database on Windows 32 bit. When I write a record, I need to retrieve the row ID / primary key so that I can a) update the record easily if I want to and b) cross reference other data to that record.
When did something similar in C, I could make a updatable cursor which allowed me to write a new record and simultaneously retrieve the row ID. With Java, it looks as though I should be able to do this, but it throws an exception with the following code.
con = openAccessDatabase();
String selectString = "SELECT ID, RunCount FROM SpeedTable";
try {
PreparedStatement selectStatement = con.prepareStatement(selectString,
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_UPDATABLE);
ResultSet idResult = selectStatement.executeQuery();
int id;
for (int i = 0; i < nWrites; i++) {
idResult.moveToInsertRow();
idResult.updateObject(1, null); // this line makes no difference whatsoever !
idResult.updateInt(2, i);
idResult.insertRow(); // throws java.sql.SQLException: [Microsoft][ODBC Microsoft Access Driver]Error in row
id = idResult.getInt(1);
}
selectStatement.close();
} catch (SQLException e) {
e.printStackTrace();
}
The only thing I've been able to do is to write a new record and then run a different query to get the Row id back ...
String insertString = "INSERT INTO SpeedTable (RunCount) VALUES (?)";
String idString = "SELECT ID FROM SpeedTable ORDER BY ID DESC";
//
try {
ResultSet idResult = null;
PreparedStatement preparedStatement, idStatement;
preparedStatement = con.prepareStatement(insertString,
ResultSet.TYPE_FORWARD_ONLY,
ResultSet.CONCUR_READ_ONLY);
idStatement = con.prepareStatement(idString,
ResultSet.TYPE_FORWARD_ONLY,
ResultSet.CONCUR_READ_ONLY);
for (int i = 0; i < nWrites; i++) {
// write the data into the database
preparedStatement.setInt(1, i);
preparedStatement.execute();
// re-run the query to get the index back from the database.
idResult = idStatement.executeQuery();
idResult.next();
int lastIndex = idResult.getInt(1);
idResult.close();
}
This works but becomes impossibly slow when the table has more than a few 10's of 1000's of records in it. There is also a risk of returning the wrong ID if two parts of the program start writing at the same time (unlikely but not impossible).
I know that at least one suggestion will be to either not use Java or not use Access, but they are not options. It's also part of a free open source software package, so I'm reluctant to pay for anything. Writing my own C JNI interface which provides the basic functionality that I need for my application is even less appealing.
Not sure if this works for MS Access but you can try:
st.executeUpdate("INSERT INTO SpeedTable (RunCount) VALUES (1000)", Statement.RETURN_GENERATED_KEYS);
ResultSet rs = st.getGeneratedKeys();
rs.next();
long id = rs.getLong(1);