My question is not related to how to do something, it's more of how to help improve performance. I apologize for the long post, but I thought since this is about performance, that i should post all details about what I'm doing to see if anyone can help.
I have to make a program that gets info from 2 different databases, creates a metadata, its corresponding BLOB(pdf file) and zips it.
The metadata file is only created if the BLOB object was found in the database. I have managed to do so, but problem is sometimes I might have up to 80k results on my query and it may take as long as 20 hours to do so, which is ridiculous considering each blob object is no larger than 100 KB.
I have a transactional database (lets call it TEQ8P) where all the ID's and the info for the metadata is stored. I queried the data by date and status (which sucks but i don't have any other filter, that was the requirement)
TEQ8P.openConnection();
Boolean flag = TEQ8P.ExecuteQuery("select tr.legaltransnumber, cc.country_code, tr.transnumber, tr.postingdate, tr.transdate from EQUATE.transheader tr inner join companycode_country cc on tr.tocompanycode = cc.company_code where tr.transtype = 'IC' and tr.transdate between to_date(" + date + ", 'DD/MM/YYYY') and to_date(" + nextday + ", 'DD/MM /YYYY')");
public Boolean ExecuteQuery(String query) {
Statement stmt;
ResultSet rs = null;
try {
stmt = connection.createStatement();
rs = stmt.executeQuery(query);
if(!rs.isBeforeFirst())
return false;
rowset = new CachedRowSetImpl();
rowset.populate(rs);
metadata = rs.getMetaData();
rs.close();
stmt.close();
return true;
} catch (SQLException e) {
HLog.error(e.getMessage());
e.printStackTrace();
return false;
//System.out.println(query);
}
finally
{
closeConnection();
}
}
I am using JAVA 1.5 (per requirement again) so I downloaded the cachedrowsetimp jar from oracle, so once I finish querying data I save it to memory and close the connection.
After that I start moving through the cachedrowset and query every ID on the Warehouse DB. I can't do a select "where in" because there is no way to tell if all id's will be found and an "in" would only return the items that it finds, and I wouldn't know which items it didn't find. But if you have any suggestion please!
So I use preparedStatement to use bind variables on ORACLE and start writing the blob object.
My first question, is there a better way to write blob files? A faster way?
if(flag)
{
String Query = "select wh.transnumber, wh.image from EQUATEWH.legalimage wh where wh.transnumber = ?";
WEQ8I.openConnection();
WEQ8I.setPreparedStmt(Query);
WEQ8I.WriteBlobs(PDF, TEQ8P.getRowsSet(), IC_FILE);
WEQ8I.closePrepStmt();
WEQ8I.closeConnection();
FileUtils.createZip(prop.getProperty("ZIPDIR_IC"), lsize, prop.getProperty("ZIPNAME_IC"));
public void WriteBlobs(String path, CachedRowSetImpl set, IMP_File IC_FILE)
{
ResultSet rs = null;
try
{
while(set.next())
{
pstmt.setString(1, set.getString(3));
rs = pstmt.executeQuery();
if(!rs.isBeforeFirst())
{
System.out.println("invoice " + set.getString(3) + "was not found on W database");
ErrorFile.writeErrorFile(set.getString(3));
}
else
{
//getting the name of the PDF file, if no ID use legaltransnumber
String ID = set.getString(1);
if(ID == null)
{
ID = set.getString(3);
}
while(rs.next())
{
FileOutputStream fos = null;
try
{
Blob blob = rs.getBlob(2);
InputStream is = blob.getBinaryStream();
fos = new FileOutputStream(path + ID + ".pdf");
int i = 0;
while ((i = is.read()) != -1)
{
fos.write(i);
}
fos.close();
is.close();
IC_FILE.fillIMPFile("IC", ID, set.getString(3), set.getString(2), set.getString(5));
}catch (Exception e)
{
e.printStackTrace();
ErrorFile.writeErrorFile(set.getString(3));
}
}
}
rs.close();
}
IC_FILE.writeFile();
} catch (SQLException e)
{
System.out.println("Problem when trying to create Record: " + path);
HLog.error(e.getMessage());
e.printStackTrace();
try
{
ErrorFile.writeErrorFile(set.getString(3));
}catch (Exception ex)
{
ex.printStackTrace();
HLog.error(e.getMessage());
}
}
}
If the query found a result for that ID on the WarehouseDB AND the image is not null (meaning it won't go to the null exception), I create the metadafile which is the IC_FILE.
the IC_FILE does not write a file, it saves everything into memory and when it finishes it writes the file, I thought this could help improve performance since it wouldn't have to do I/O operations on every file, just once, using IC_FILE.writefile().
To create the metadatafile, I also have to (per requirement again) get the container name from a file. to retrieve the container name I have to use 3 fields from the Transactional database, concatenate them and search for them in that file.
this is how I create the IMP file, first to get data from each record:
public void fillIMPFile(String type, String ID, String ID2, String companyCode, String date)
{
date = date.substring(0, 10);
date = date.replace("-", "/");
date = date.substring(5, 7) + "/" + date.substring(8, 10) + "/" + date.substring(0, 4);
String Name = prop.getProperty("NAME");
String info = prop.getProperty(type);
String DOS = Name + info + ID + ".";
String NOTES = Name + " " + info + " ";
info += getContainer(companyCode, date, type);
if(type.equals("IC"))
{
String desc = prop.getProperty("DESC_PDF");
DOS += "pdf";
NOTES += desc + " " + ID + " " + ID2;
buffer += info + "\t" + date + "\t" + date + "\t" + DOS + "\t" + NOTES + "\t"
+ NOTES + "\t" + ID2;
}
To get the containers I use the properties object, but I guess there might be better choices? a hash map maybe?
public String getContainer(String companyCode, String Date, String type)
{
Calendar cal = Calendar.getInstance();
SimpleDateFormat sdf = new SimpleDateFormat("yyyy");
Date = sdf.format(cal.getTime());
//mal siempre pondra 2012
String data = type + companyCode + Date;
String container = containers.getProperty(data);
if(container == null)
{
data = type + "WW" + Date;
container = containers.getProperty(data);
}
return container;
}
finaly to write the file:
public void writeFile()
{
try
{
FileWriter fw = new FileWriter(File, true);
BufferedWriter bw = new BufferedWriter(fw);
bw.write(buffer);
bw.close();
}catch (IOException e)
{
e.printStackTrace();
HLog.error(e.getMessage());
}
}
thanks!!
Daniel
fixed...using bind variables and creating more than 1 container file for each year
Related
I am taking input from user and storing in two different variables. I am binding the parameters with my sql statement. When i run the code its giving issue on concatenating part of query.
String CityA= null;
String CityB= null;
try {
CityA = readEntry(in, "Enter Origin City : ");
CityB = readEntry(in, "Enter Destination City : ");
// We treat this drop table specially to allow it to fail
// as it will the very first time we run this program
try {
String q = "SELECT f.FLNO,f.DISTANCE,TIMEDIFF(f.arrives,f.departs)
as Duration FROM FLIGHTS F"
+ " WHERE F.ORIGIN = "+CityA;
+ "AND f.DESTINATION = "+CityB;
System.out.println(q);
rset = stmt.executeQuery(q);
while (rset.next()) {
System.out.println(rset.getInt("FLNO") + ","
+ rset.getInt("Distance") + ","
+ rset.getTime("Duration"));
}
System.out.println("Done");
}
catch (SQLException e) {
// assume not there yet, so OK to continue
}
finally {
stmt.close();
}
Please find the code for query:-
Basically you missed the space between the CityA and AND
String q = "SELECT f.FLNO,f.DISTANCE,TIMEDIFF(f.arrives,f.departs) as Duration FROM FLIGHTS F"
+ " WHERE F.ORIGIN = '"+CityA+"' ";
+ "AND f.DESTINATION = '"+CityB+"'";
There is a typo in your query string - you missed the space between 'Los-Angeles' and AND.
I'm trying to create an api so I can use it to whatever project I need to create. I'm still new to java so I'm sorry in advance for the wrong codes. Anyways, I've got here this code that has the CRUD statements (except for Read/Retrieve).
import java.sql.*;
public class KitApiNew {
public static void main(String[] args) throws SQLException {
Connection dbConnection = null;
PreparedStatement PSInsert = null;
PreparedStatement PSUpdate = null;
PreparedStatement PSDelete = null;
PreparedStatement PSStatement = null;
String DBTable = "fruits";
String DBColumnSet = "fruit";
String DBID = "23";
String DBColumnSingle = "fruit";
String DBChoice ="insert";
String DBCS = "";
String insertTable = "INSERT INTO " + DBTable + "(" +DBColumnSet+ ")" + "VALUES" + "(?)";
String updateTable = "UPDATE " + DBTable + " SET " + DBColumnSingle + " = ?" + " WHERE id = ?";
String deleteTable = "DELETE FROM " + DBTable + " WHERE id = ?";
String statementTable = "INSERT INTO fruits(fruit) VALUES('grapes')";
try{
dbConnection = getDBConnection();
dbConnection.setAutoCommit(false);
if(DBChoice.equals("insert")){
//for insert
PSInsert = dbConnection.prepareStatement(insertTable);
PSInsert.setString(1, "Orange");
PSInsert.executeUpdate();
dbConnection.commit();
}
if(DBChoice.equals("update")){
//for update
PSUpdate = dbConnection.prepareStatement(updateTable);
PSUpdate.setString(1, "Apple");
PSUpdate.setString(2, DBID);
PSUpdate.executeUpdate();
dbConnection.commit();
}
if(DBChoice.equals("delete")){
//for delete
PSDelete = dbConnection.prepareStatement(deleteTable);
PSDelete.setString(1, DBID);
PSDelete.executeUpdate();
dbConnection.commit();
}
if(DBChoice.equals("statement")){
//for statement
PSStatement = dbConnection.prepareStatement(statementTable);
PSStatement.executeUpdate();
dbConnection.commit();
}
System.out.println("Success!");
}catch(SQLException e){
System.out.println("Error occured " + e.toString());
dbConnection.rollback();
}
finally{
if(PSInsert !=null){
PSInsert.close();
}
if(PSUpdate != null){
PSUpdate.close();
}
if(dbConnection != null){
dbConnection.close();
}
}
}
private static Connection getDBConnection(){
Connection con = null;
try{
Class.forName("com.mysql.jdbc.Driver");
}catch(ClassNotFoundException e){
System.out.println("Error 1 : " + e.getMessage());
}
try{
con = DriverManager.getConnection("jdbc:mysql://localhost:3306/bongbong","root","");
}catch(SQLException e){
System.out.println("Error 2 : " + e.getMessage());
}
return con;
}
}
Oh by the way,
The values of the variables are currently temporary. I'll be changing them later to whatever they're going to be catching. I'm referring to these:
String DBTable = "fruits";
String DBColumnSet = "fruit";
String DBID = "23";
String DBColumnSingle = "fruit";
String DBChoice ="insert";
String DBCS = "";
and this
String statementTable = "INSERT INTO fruits(fruit) VALUES('grapes')";
Problem
What I really need help from is with the ? thing from the first code I posted after the VALUES word inside the String insertTable. I want it so that when I place a column amount in my interface then it'll add more ? in accordance with what was inputted (I also want it to add another PSInsert.setString(n, "value") with n+1 in accordance with the column amount inputted if it's possible). Can anyone tell me how to? I'm really new to java and I'm still a student studying at his best.
I want it to add those ? thing because what if I add more columns or if I use another table with more columns other than my fruits table. (I want it so that whatever I'm going to place in DBColumnSet --with columns separated by comma --will also relate to how many ? are going to be placed).
Oh by the way, it's a general api so I can't provide an interface.
I'm trying to retrieve all the values stored in a database where it matches the sensor name.
The code runs however, it only outputs the final value in the database.
String info = "test info"
String sensorNameStr = "test sensor name";
String sensorValueStr = "test sensor value";
String selectSQL = "select sensorvalue from sensorUsage where "+
" sensorname = '" + sensorNameStr +
"' order by TimeInserted asc";
String retrievedSensorData = "no data available";
try {
PreparedStatement pst = conn.prepareStatement(selectSQL);
ResultSet rs = pst.executeQuery();
while (rs.next()) {
retrievedSensorData = rs.getString(1);
}
} catch (SQLException ex) {
System.out.println("Error in SQL " + ex.getMessage());
}
String json = "{\"sensor\": {\"" + sensorNameStr +
"\": \"" + retrievedSensorData + "\"}}";
System.out.println("DEBUG: json return: "+json);
Do I have to create a for loop so all values are returned and outputted on a webpage? As that's how I am viewing the data without manually going into the server. Also, if I do this, is it possible to only receive the last few inserted sensor values?
Help will be appreciated!
In your SQL query you retrieve only rows from column sensorvalue and all belong to specific sensorname. There are no other condition.
So it should be easy to make an Json with the Java EE API
// ...
JsonObject model = null;
try
{
// ...
ResultSet rs = stmt.executeQuery();
JsonArrayBuilder jsonArrayBuilder = Json.createArrayBuilder();
while(rs.next())
{
jsonArrayBuilder.add(rs.getString("sensorvalue"));
}
JsonObjectBuilder jsonObjectBuilder = Json.createObjectBuilder();
jsonObjectBuilder.add("sensorvalues", jsonArrayBuilder);
model = jsonObjectBuilder.build();
}
catch (Exception e)
{
System.out.println(e);
}
response.setContentType("application/json; charset=utf-8");
StringWriter stWriter = new StringWriter();
JsonWriter jsonWriter = Json.createWriter(stWriter);
jsonWriter.writeObject((JsonObject) model);
jsonWriter.close();
response.getWriter().append(stWriter.toString());
The output should be something like:
{"sensorvalues":["1","2"]}
I have a problem with a really slow connection between my Java code and a MySQL remote Database when i use multiple query.
This is my code
ArrayList<Server_Log> ar =Server_Log_Utilities.getBy2Dates(cmb_date.getSelectedItem() + "", cmb_date2.getSelectedItem() + "");
for (int c = 0; c < ar.size(); c++) {
Server_Log sl = ar.get(c);
String username = User_Utilities.getUserName(sl.getUser() + "");
String row[] = {sl.getDate(), sl.getTime(), username, sl.getReff(), sl.getDescription()};
}
but I user this code data will load fast
ArrayList<Server_Log> ar =Server_Log_Utilities.getBy2Dates(cmb_date.getSelectedItem() + "", cmb_date2.getSelectedItem() + "");
for (int c = 0; c < ar.size(); c++) {
Server_Log sl = ar.get(c);
String row[] = {sl.getDate(), sl.getTime(), sl.getReff(), sl.getDescription()};
}
this is User_Utilities.getUserName(sl.getUser() + ""); Method
public static String getUserName(String id) {
String UserName="";
try {
Connection con = new DBCon().getConnection();
ResultSet rst = DBHandle.getData(con, "SELECT username FROM user WHERE id='" + id + "'");
while (rst.next()) {
UserName =rst.getString(1);
}
con.close();
} catch (Exception ex) {
Logger.getLogger(User_Utilities.class.getName()).log(Level.SEVERE, null, ex);
}
return UserName;
}
Server_Log_Utilities.getBy2Dates(cmb_date.getSelectedItem() + "",
cmb_date2.getSelectedItem() + ""); Method
public static ArrayList getBy2Dates(String date1, String date2) {
try {
ar = new ArrayList<>();
Connection con = new DBCon().getConnection();
ResultSet rst = DBHandle.getData(con, "SELECT * FROM server_log WHERE date BETWEEN '" + date1 + "' AND '" + date2 + "' ORDER BY `date`");
while (rst.next()) {
Server_Log ci = new Server_Log();
ci.setId(rst.getInt(1));
ci.setDate(rst.getString(2));
ci.setTime(rst.getString(3));
ci.setReff(rst.getString(4));
ci.setDescription(rst.getString(5));
ci.setUser(rst.getInt(6));
ar.add(ci);
}
con.close();
} catch (Exception ex) {
Logger.getLogger(Student_Utilities.class.getName()).log(Level.SEVERE, null, ex);
}
return ar;
}
When accessing a remote database, especially over a slow link, the number of SQL statements executed is very important.
This is why the JDBC API support concepts like statement batching.
In your case, you're calling getUserName for every record in ar. Consider ways to reduce the number of calls.
Example 1: If user is usually the same, or only a few users are generating log entries, caching the user names would eliminate redundant lookup.
Example 2: Rather than looking up the user in the client, modify the Server_Log_Utilities.getBy2Dates to add a JOIN to the User table. This way, no extra turn-arounds to database will be needed.
Example 3: Instead of calling getUserName individually in a loop, collect the user ids, and lookup the names in a batch. Use either a JDBC batch of multiple SELECT statements, or use a single statement with UserId IN (?,?,?,?,...).
I am using apache velocity template to format the content then send the mails. But template content will be availabe in database table. For this we can use DataSourceResourceLoader.
But it works with predefined database table format and it gets only templateContent. But i want to get few more columns from the same table using the same DataSourceResourceLoader.
How can i do that?
Please find below code from DataSourceResourceLoader where it gets only one column.
public synchronized InputStream getResourceStream(final String name)
throws ResourceNotFoundException
{
if (org.apache.commons.lang.StringUtils.isEmpty(name))
{
throw new ResourceNotFoundException("DataSourceResourceLoader: Template name was empty or null");
}
Connection conn = null;
ResultSet rs = null;
PreparedStatement ps = null;
try
{
conn = openDbConnection();
ps = getStatement(conn, templateColumn, tableName, keyColumn, name);
rs = ps.executeQuery();
if (rs.next())
{
InputStream stream = rs.getBinaryStream(templateColumn);
if (stream == null)
{
throw new ResourceNotFoundException("DataSourceResourceLoader: "
+ "template column for '"
+ name + "' is null");
}
return new BufferedInputStream(stream);
}
else
{
throw new ResourceNotFoundException("DataSourceResourceLoader: "
+ "could not find resource '"
+ name + "'");
}
}
catch (SQLException sqle)
{
String msg = "DataSourceResourceLoader: database problem while getting resource '"
+ name + "': ";
log.error(msg, sqle);
throw new ResourceNotFoundException(msg);
}
catch (NamingException ne)
{
String msg = "DataSourceResourceLoader: database problem while getting resource '"
+ name + "': ";
log.error(msg, ne);
throw new ResourceNotFoundException(msg);
}
finally
{
closeResultSet(rs);
closeStatement(ps);
closeDbConnection(conn);
}
}
In above code InputStream stream = rs.getBinaryStream(templateColumn); is retrieved and returned. Now i need to get few more columns. How can i do that?
Thanks!