I've got database querying that has become too slow with my current implementation. I need to get all movies from a database and for each of these movies i need their file data from another table. So for each movie i am doing another query. For each candidate entry i need to do a comparison to every movie in the database. Should this be taking 5-10 seconds to execute for approximately 500 candidates?
// get movies with all their versions
private ArrayList<Movie> getDatabaseMovies(Connection conn) throws Exception {
PreparedStatement getMoviesStmt = conn.prepareStatement("SELECT movieid, title FROM movies", Statement.RETURN_GENERATED_KEYS);
ArrayList<Movie> movies = new ArrayList<Movie>();
try {
ResultSet rs = getMoviesStmt.executeQuery();
while (rs.next()) {
Movie movie = new Movie(rs.getString(2), getDatabaseMovieFiles(conn, rs.getInt(1)));
movies.add(movie);
}
} catch (SQLException e) {
e.printStackTrace();
} finally {
getMoviesStmt.close();
}
return movies;
}
public ArrayList<MovieFile> getDatabaseMovieFiles(Connection conn, int movieID) throws Exception {
ArrayList<MovieFile> movieFiles = new ArrayList<MovieFile>();
PreparedStatement stmt = conn.prepareStatement("SELECT filename, size, hash, directory FROM file_video WHERE movieid = ?");
try {
stmt.setInt(1, movieID);
ResultSet rs = stmt.executeQuery();
while (rs.next()) {
MovieFile movieFile = new MovieFile(rs.getString(1), rs.getLong(2), rs.getBytes(3), rs.getString(4));
movieFiles.add(movieFile);
}
} catch (SQLException e) {
e.printStackTrace();
} finally {
stmt.close();
}
return movieFiles;
}
Should this be taking 5-10 seconds to execute for approximately 500 candidates?
Probably not.
There are two ways to improve this:
Make sure that there is an index on the movieid column of file_video.
Combine the two queries into one by using a JOIN.
You probably should do both.
Related
I have:
Map<String, ExtractData> extractMap = new HashMap<String, ExtractData>();
Why would the following method not get return data from the method it calls?
private void fetchExtractData(CentralExportSession exportSession) throws RemoteException {
System.out.println("Fetching Extract Data");
extractMap = exportSession.getExtractData(consDB);
System.out.printLn("Extract Data Fetched");
}
This calls into:
public Map<String, ExtractData> getExtractData(DataSourceInfo ds) throws RemoteException {
Connection conn = null;
PreparedStatement ps = null;
ResultSet rs = null;
Map<String, ExtractData> extractMap = new HashMap<String, ExtractData>();
String qry = "...select statement that returns about 500k rows...;"
try {
conn = getConnection();
ps = conn.PrepareStatment(qry);
rs = ps.executeQuery();
while (rs.next()) {
String dataKey = rs.getString("key");
ExtractData data = new ExtractData();
...code that builds up the 'data' object from the result...
extractMap.put(dataKey, data);
System.out.println("Added key : " + dataKey);
}
} catch (SQLException e) {
e.printStackTrace();
throw new EJBException(e);
} finally {
System.out.println("Cleaning up...");
cleanup(conn, ps, rs);
System.out.println("Returning Map extractMap...");
return extractMap;
}
}
The "Fetching Extract Data" prints, as do the "Added Key :", "Cleaning up..." and "Returning Map extractMap..." but not the "Extract Data Fetched" in the calling method. I get no visible exceptions, control just is never passed back to the calling method and my CPU usage goes through the roof until I kill it.
Edit: I've tried moving the "return" statement to both inside the "try" block (after the "while") and after the "finally" block with no difference. The location of the return statement does not affect the (lack of) outcome.
It seems to be a memory issue as artificially limiting the resultset to 200k rows allows it to run, but once I go much past that...
The result set that I'm trying to retrieve from another class returns null, even though the query works.I'm trying to initialize my object based on the records kept in databases,which means if there is initially a record in sqlite,I retrieve the one with latest date.Else,I try to retrieve the earliest one from mysql database. The code that is supposed to retrieve result set from mysql database is like this:
public ResultSet lowestDate() throws SQLException {
ResultSet rs1 = null;
String resultQuery = "SELECT * FROM alarm ORDER BY `timestamp` ASC LIMIT 1";
rs1 = stmt.executeQuery(resultQuery);
return rs1;
}
Statement is initialized globally.And I call this in another class like this:
public void setLastAlarm() throws SQLException, ParseException {
String liteQuery = "SELECT * FROM alarm_entries ORDER BY date(`timestamp`) DESC LIMIT 1";
conn.connectLite();
Connection getCon = conn.getLiteConnection();
try {
stmt = getCon.createStatement();
} catch (SQLException e) {
e.printStackTrace();
}
try {
rs = stmt.executeQuery(liteQuery);
if (rs.next()) {
//while (rs.next()) {
nuDate = rs.getString("timestamp");
newDate = format.parse(nuDate);
lastAlarm.setBacklogId(rs.getBytes("backlog_id"));
lastAlarm.setTimestamp(newDate);
//}
}
else{
rsq=mysqlConnection.lowestDate();
lastAlarm.setTimestamp(format.parse(rsq.getString("timestamp")));
lastAlarm.setBacklogId(rsq.getBytes("backlog_id"));
}
}catch (Exception e){
e.printStackTrace();
}
}
public void run() {
try {
setLastAlarm();
You never call ResultSet#next() on the result set being returned from the lowestDate() helper method. Hence, the cursor is never being advanced to the first (and only) record in the result set. But I think it is a bad idea to factor your JDBC code in this way. Instead, just inline your two queries like this:
try {
rs = stmt.executeQuery(liteQuery);
if (rs.next()) {
nuDate = rs.getString("timestamp");
newDate = format.parse(nuDate);
lastAlarm.setBacklogId(rs.getBytes("backlog_id"));
lastAlarm.setTimestamp(newDate);
}
else {
String resultQuery = "SELECT * FROM alarm ORDER BY timestamp LIMIT 1";
rs = stmt.executeQuery(resultQuery);
if (rs.next()) {
String ts = rs.getString("timestamp");
lastAlarm.setTimestamp(format.parse(ts));
lastAlarm.setBacklogId(rs.getBytes("backlog_id"));
}
}
} catch (Exception e){
e.printStackTrace();
}
When Try to fetching Data in database result it come zero row but when try to copy and past query on mysql has return specific number of rows needed.
Connection to mysql server
private Connection connection() {
try {
Class.forName("com.mysql.jdbc.Driver").newInstance();
Connection con = DriverManager.getConnection("jdbc:mysql://localhost:3306/olesdb", "root", "");
} catch (Exception e) {
//System.out.println("Connection error");
}
return con;
}
**
My function for fetching data
**
public List<AcademicYearCourses> getStudentCourse(int studentID, int academicYear,int semester) throws SQLException{
List<AcademicYearCourses> list = new ArrayList<>();
PreparedStatement sta = connection().prepareStatement("SELECT courseCode,courseName FROM courses co,studentprograms stpro,academicyearcourse acco WHERE stpro.studentID=? AND acco.academicYearID=? AND acco.semesterID=? AND stpro.programID= acco.programID AND stpro.studyYear=acco.studyYear AND acco.courseID=co.courseID");
sta.setInt(1, studentID);
sta.setInt(2, academicYear);
sta.setInt(3, semester);
ResultSet res = sta.executeQuery();
while(res.next()){
AcademicYearCourses acco = new AcademicYearCourses();
acco.setAcdemicYearCourseID(rs.getInt("acdemicYearCourseID"));
acco.setCourseName(rs.getString("courseName"));
acco.setCourseCode(rs.getString("courseCode"));
list.add(acco);
}
return list;
}
So I need help to solve this issue it very important in my project and Cant continue without this data
Your are doing rs.getInt("acdemicYearCourseID") but column acdemicYearCourseID is not in your SELECT columns list.
Also try changing getInt("..."), getString("...") to getInt(1), getString(2)
I need help with the code below and getting it to return a true or false value. Any and all help would be appreciated.
public synchronized static boolean checkCompanyName(String companyName,
Statement statement) {
try {
ResultSet res = statement
.executeQuery("SELECT `companyName` FROM `companys` WHERE companyName = '"
+ companyName + "';");
boolean containsCompany = res.next();
res.close();
return containsCompany;
} catch (Exception e) {
e.printStackTrace();
return false;
}
}
Try to make your query like this:
ResultSet res = statement.executeQuery("SELECT companyName FROM companys WHERE companyName = " + companyName);
Or you can either you PreparedStatement which is better then you did before
You should be using a PreparedStatement (for that end pass the Connection in to the method). Also, you should retrieve the value from the ResultSet and validate it matches your companyName. Something like
static final String query = "SELECT `companyName` FROM "
+ "`companys` WHERE companyName = ?";
public synchronized static boolean checkCompanyName(String companyName,
Connection conn) {
PreparedStatement ps = null;
ResultSet rs = null;
try {
ps = conn.prepareStatement(query);
ps.setString(1, companyName);
rs = ps.executeQuery();
if (rs.next()) {
String v = rs.getString(1);
return v.equals(companyName);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (rs != null) {
try {
rs.close();
} catch (SQLException e) {
}
}
if (ps != null) {
try {
ps.close();
} catch (SQLException e) {
}
}
}
return false;
}
Two comments:
You only need to check if there's at least one row matching your criteria, so you can use .first()
Your code is vulnerable to SQL Injection attacks. Please read this to learn more about it.
The easiest way to avoid SQL injection attacs is to use prepared statements. So let me strike two birds with a single stone and give you a solution using them:
/*
Check if the company exists.
Parameters:
conn - The connection to your database
company - The name of the company
Returns:
true if the company exists, false otherwise
*/
public static boolean checkCompanyName(Connection conn, String company) {
boolean ans = false;
try(PreparedStatement ps = conn.prepareStatement(
"select companyName from companies where companyName = ?"
) // The question mark is a place holder
) {
ps.setString(1, company); // You set the value for each place holder
// using setXXX() methods
try(ResultSet rs = ps.executeQuery()) {
ans = rs.first();
} catch(SQLException e) {
// Handle the exception here
}
} catch(SQLException e) {
// Handle the exception here
}
return ans;
}
Suggested reads:
Bobby Tables: A guide to preventing SQL injection
The Java Tutorials - JDBC: Using prepared statements
I have a problem trying to execute more than one query into my Java Application code.
I have a procedure that is called in main and is in the class "Fant":
public void XXX(){
Connectivity con=new Connectivity(); // this class set up the data for the connection to db; if ( !con.connect() ) {
System.out.println("Error during connection.");
System.out.println( con.getError() );
System.exit(0);
}
ArrayList<User> blabla=new ArrayList<User>();
blabla=this.getAllUsers(con);
for (User u:blabla)
{
try {
Connectivity coni=new Connectivity();//start a new connection each time that i perform a query
Statement t;
t = coni.getDb().createStatement();
String query = "Select count(*) as rowcount from berebe.baraba";
ResultSet rs = t.executeQuery(query);
int numPrenotazioni=rs.getInt("rowcount");
rs.close(); //close resultset
t.close(); //close statement
coni.getDb().close(); //close connection
}
}
catch (SQLException e)
{
System.err.println("SQLState: " +
((SQLException)e).getSQLState());
System.err.println("Error Code: " +
((SQLException)e).getErrorCode());
}
}
}
The called function is defined as:
ArrayList<User> getAllUsers(Connectivity con) {
try{
ArrayList<User> userArrayList=new ArrayList<User>();
String query = "Select idUser,bubu,lala,sisi,gogo,gg from berebe.sasasa";
Statement t;
t = con.getDb().createStatement();
ResultSet rs = t.executeQuery(query);
while (rs.next())
{
User utente=new User(....); //user fields got from query
userArrayList.add(utente);
}
rs.close();
t.close();
con.disconnect(); //disconnect the connection
return userArrayList;
} catch (SQLException e) {
}
return null;
}
The main is:
public static void main(String[] argv) {
ArrayList<User> users=new ArrayList<User>();
System.out.println("-------- MySQL JDBC Connection Testing ------------");
Fant style = new Fant();
style.XXX();
}
The query performed into "getAllusers" is executed and into the arraylist "blabla" there are several users; the problem is that the second query that needs the count is never executed.
The MYSQlState given when running is= "S1000" and the SQLERROR is "0".
Probably i'm mistaking on connections issues but i'm not familiar with statements,connections,resultsets.
Thank you.
You might forget to call rs.next() before getting the result form it in XXX()methods as shown below:
ResultSet rs = t.executeQuery(query);
// call rs.next() first here
int numPrenotazioni=rs.getInt("rowcount");