I have:
Map<String, ExtractData> extractMap = new HashMap<String, ExtractData>();
Why would the following method not get return data from the method it calls?
private void fetchExtractData(CentralExportSession exportSession) throws RemoteException {
System.out.println("Fetching Extract Data");
extractMap = exportSession.getExtractData(consDB);
System.out.printLn("Extract Data Fetched");
}
This calls into:
public Map<String, ExtractData> getExtractData(DataSourceInfo ds) throws RemoteException {
Connection conn = null;
PreparedStatement ps = null;
ResultSet rs = null;
Map<String, ExtractData> extractMap = new HashMap<String, ExtractData>();
String qry = "...select statement that returns about 500k rows...;"
try {
conn = getConnection();
ps = conn.PrepareStatment(qry);
rs = ps.executeQuery();
while (rs.next()) {
String dataKey = rs.getString("key");
ExtractData data = new ExtractData();
...code that builds up the 'data' object from the result...
extractMap.put(dataKey, data);
System.out.println("Added key : " + dataKey);
}
} catch (SQLException e) {
e.printStackTrace();
throw new EJBException(e);
} finally {
System.out.println("Cleaning up...");
cleanup(conn, ps, rs);
System.out.println("Returning Map extractMap...");
return extractMap;
}
}
The "Fetching Extract Data" prints, as do the "Added Key :", "Cleaning up..." and "Returning Map extractMap..." but not the "Extract Data Fetched" in the calling method. I get no visible exceptions, control just is never passed back to the calling method and my CPU usage goes through the roof until I kill it.
Edit: I've tried moving the "return" statement to both inside the "try" block (after the "while") and after the "finally" block with no difference. The location of the return statement does not affect the (lack of) outcome.
It seems to be a memory issue as artificially limiting the resultset to 200k rows allows it to run, but once I go much past that...
Related
In my JavaFX program I want to dynamically generate a tree using the live data set from my database (I'm using MariaDB, but it could be any SQL database).
I had searched a bunch and could not find a direct answer to my question, so I spent some time learning how JDBC ResultSet's work, how the next() method works, and how while loops work. A few trial-and-error attempts finally led me to the result I wanted, so I thought I would share it in case anyone else finds themselves in my position.
See my answer below.
edit It seems I’ve designed the overall program poorly in not using threading to isolate the GUI from the JDBC query. I think this can be fixed using JavaFX concurrency, but I’ve never used it so until I can update the code below, just ignore the stuff outside the while loop.
Firstly, I'm using following versions of MariaDB (10.4.12), Java (13.0.2.8), JavaFX (13.0.2), and MariaDB JDBC (2.6.0). I don't think any of this will make a difference, but just in case.. I'm using FXML, which is why you won't see any UI formatting in there anywhere.
This is the full method that generates the TreeView in my JavaFX program. It's then called from a separate class shortly after generating the Stage and Scene objects.
They key part is the while loop, which was a struggle for me to get right. I initially thought I needed a nested while(rs.next()) loop, but then I realised that this was causing rows to be skipped because each rs.next() call is related to the previous one, not to the while loop in which it is used.
Note also that the SQL statement is important. If the statement gives results out of order the method doesn't work correctly.
public void generateTree(DataSource dataSource) {
source = null;
this.source = dataSource;
Connection con = null;
Statement stmt = null;
TreeItem<String> rootTreeItem = new TreeItem<String>("EMT");
rootTreeItem.setExpanded(true);
emtTree.setRoot(rootTreeItem);
TreeItem<String> site = null;
TreeItem<String> plant = null;
try {
con = source.getConnection();
try {
stmt = con.createStatement();
ResultSet rs = stmt.executeQuery("SELECT sites.siteid, sites.longname, plants.plantid, plants.siteplantid, plants.shortname FROM sites INNER JOIN plants ON plants.siteid=sites.siteid ORDER BY sites.longname ASC, plants.siteplantid ASC");
String site1 = ""; //It's not possible for the site name to be "" in the result set because the database design prevents it.
//Steps through the result set from first row item to last row item.
while(rs.next()) {
//This bit prevents repeating the same first level items multiple times.
//I only want each site to appear once, and under each site is a list
//of plants.
if (!site1.equals(rs.getString("sites.longname"))) {
site = new TreeItem<String>(rs.getString("sites.longname"));
rootTreeItem.getChildren().add(site);
site1 = rs.getString("sites.longname");
}
//This section is never skipped and will add all the plants to a given
//site until the next site is reached in the result set, then the if
//is triggered again and the process for the new site.
plant = new TreeItem<String>(rs.getInt("plants.siteplantid") + " " + rs.getString("plants.shortname"));
site.getChildren().add(plant);
}
} catch (SQLException e) {
System.out.println("Statement Error");
System.err.println("SQL State: " + ((SQLException)e).getSQLState());
System.err.println("Error Code: " + ((SQLException)e).getErrorCode());
System.err.println("Message: " + ((SQLException)e).getMessage());
System.err.println("Cause: " + ((SQLException)e).getCause());
return;
}
} catch (SQLException e) {
e.printStackTrace();
return;
} finally {
if (stmt != null) {
try {
stmt.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
if (con != null) {
try {
con.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
}
I'll probably find ways to improve it in future, but for now I'm just happy to have it working. Ignore the messy exception handling - its a work in progress!
Here is my try how to do it with not blocking the UI Thread:
public class YourClassName {
ArrayList<ResultBean> fetchedData = null;
public void createTree() {
// run in other thread fetching the data
Task task = new Task<Void>() {
#Override
protected Void call() throws Exception {
// show loading pane -> you can use the one from import org.controlsfx.control.MaskerPane
// https://github.com/controlsfx/controlsfx/wiki/ControlsFX-Features
loadingPane.setVisible(true);
// get data
fetchedData = generateTreeData(yourDataSource);
return null;
}
#Override
protected void succeeded() {
// the data has been fetched, now is safe to build your tree
super.succeeded();
buildTreeFromTheData(fetchedData);
}
};
new Thread(task).start();
}
}
Fetch the data from your database:
public ArrayList<ResultBean> getTreeData(DataSource dataSource) {
ArrayList<ResultBean> resultBeans = new ArrayList<>();
source = null;
this.source = dataSource;
Connection con = null;
Statement stmt = null;
try {
con = source.getConnection();
try {
stmt = con.createStatement();
ResultSet rs = stmt.executeQuery("SELECT " +
"sites.siteid, " +
"sites.longname, " +
"plants.plantid, " +
"plants.siteplantid, " +
"plants.shortname " +
"FROM sites " +
"INNER JOIN plants ON plants.siteid=sites.siteid " +
"ORDER BY sites.longname ASC, plants.siteplantid ASC");
while (rs.next()) {
ResultBean resBean = new ResultBean();
resBean.setSiteId(rs.getString("sites.siteid"));
//....
//....
// set all values and add it to the result array
resultBens.add(resBean);
}
} catch (SQLException e) {
e.printStackTrace();
}
} catch (SQLException e) {
e.printStackTrace();
} finally {
if (stmt != null) {
try {
stmt.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
if (con != null) {
try {
con.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
return resultBens;
}
ResultBean:
public class ResultBean {
private int siteid;
private String longname;
private int plantid;
private int siteplantid;
private String shortname;
// setters
....
// getters
....
}
And finally, build the tree with your logic:
public void buildTreeFromTheData(ArrayList<ResultBean> treeData) {
// do you logic here, loop over treeData ArrayList in while loop
TreeItem<String> rootTreeItem = new TreeItem<String>("EMT");
rootTreeItem.setExpanded(true);
emtTree.setRoot(rootTreeItem);
TreeItem<String> site = null;
TreeItem<String> plant = null;
//This bit prevents repeating the same first level items multiple times.
//I only want each site to appear once, and under each site is a list
//of plants.
//if (!site1.equals(rs.getString("sites.longname"))) {
// site = new TreeItem<String>(rs.getString("sites.longname"));
// rootTreeItem.getChildren().add(site);
// site1 = rs.getString("sites.longname");
// }
//This section is never skipped and will add all the plants to a given
//site until the next site is reached in the result set, then the if
//is triggered again and the process for the new site.
// plant = new TreeItem<String>(rs.getInt("plants.siteplantid") + " " + rs.getString("plants.shortname"));
// site.getChildren().add(plant);
// finally, hide the loading pane
maskerPane.setVisible(false);
}
I'm trying to execute method which should create a new object with fields from database, and everytime i run this code im getting SQLException: ResultSet closed.
public DatabasedClient getDatabaseClient(int clientDatabaseid){
if(DatabaseClientUtil.isInDatabase(clientDatabaseid)){
return DatabaseClientUtil.getDBClient(clientDatabaseid);
}else{
try{
System.out.println("Trying to find user in db");
ResultSet rs = fbot.getStorage().query("select * from database_name where clientDBId = " + clientDatabaseid);
System.out.println("deb " + rs.getString("nick"));
while (rs.next()) {
DatabasedClient databasedClient = new DatabasedClient(clientDatabaseid);
databasedClient.setUid(rs.getString("uid"));
databasedClient.setNick(rs.getString("nick"));
databasedClient.setLastConnect(rs.getLong("lastConnected"));
databasedClient.setLastDisconnect(rs.getLong("lastDisconnect"));
databasedClient.setTimeSpent(rs.getLong("timeSpent"));
databasedClient.setLongestConnection(rs.getLong("longestConnection"));
return databasedClient;
}
} catch (SQLException e) {
e.printStackTrace();
}
}
return null;
}
}
Im using hikari, here are methods from AbstractStorage class
#Override
public void execute(String query) throws SQLException {
try (Connection connection = getConnection()){
connection.prepareStatement(query).executeUpdate();
}
}
#Override
public ResultSet query(String query) throws SQLException {
try (Connection connection = getConnection()) {
return connection.prepareStatement(query).executeQuery();
}
}
Screenshot from error
I hope someone will help me with this.
I think the exact error you are seeing is being caused by the following line of code:
System.out.println("deb " + rs.getString("nick"));
You are trying to access the result set before you advance the cursor to the first record. Also, your method getDatabaseClient is returning a single object which conceptually maps to a single expected record from the query. Hence, iterating once over the result set would seem to make sense. Taking all this into consideration, we can try the following:
try {
System.out.println("Trying to find user in db");
ResultSet rs = fbot.getStorage().query("select * from database_name where clientDBId = " + clientDatabaseid);
// do not access the result set here
if (rs.next()) {
DatabasedClient databasedClient = new DatabasedClient(clientDatabaseid);
databasedClient.setUid(rs.getString("uid"));
databasedClient.setNick(rs.getString("nick"));
databasedClient.setLastConnect(rs.getLong("lastConnected"));
databasedClient.setLastDisconnect(rs.getLong("lastDisconnect"));
databasedClient.setTimeSpent(rs.getLong("timeSpent"));
databasedClient.setLongestConnection(rs.getLong("longestConnection"));
return databasedClient;
}
} catch (SQLException e) {
e.printStackTrace();
}
first time posting so sorry if my question is slightly strange.
So I have a project in school that requires us to create java classes using netbeans that open up a window with three options, check stock, purchase item and update stock.
We had a class called stockdata that held the details of 5 different items for us to use in our three classes to check, purchase and update items. The latest stage of our coursework requires us to create a derby database and enter the items into a table.
I have done this with no issues but I am having a problem getting the items from the table back into my classes to use. We were given the following code but I can't get it to work, even using the commented hints.
package stock;
// Skeleton version of StockData.java that links to a database.
// NOTE: You should not have to make any changes to the other
// Java GUI classes for this to work, if you complete it correctly.
// Indeed these classes shouldn't even need to be recompiled
import java.sql.*; // DB handling package
import java.io.*;
import org.apache.derby.drda.NetworkServerControl;
public class StockData {
private static Connection connection;
private static Statement stmt;
static {
// standard code to open a connection and statement to an Access database
try {
NetworkServerControl server = new NetworkServerControl();
server.start(null);
// Load JDBC driver
Class.forName("org.apache.derby.jdbc.EmbeddedDriver");
//Establish a connection
String sourceURL = "jdbc:derby://localhost:1527/"
+ new File("UserDB").getAbsolutePath() + ";";
connection = DriverManager.getConnection(sourceURL, "use", "use");
stmt = connection.createStatement();
} // The following exceptions must be caught
catch (ClassNotFoundException cnfe) {
System.out.println(cnfe);
} catch (SQLException sqle) {
System.out.println(sqle);
} catch (Exception e) {
System.out.println(e);
}
}
// You could make methods getName, getPrice and getQuantity simpler by using an auxiliary
// private String method getField(String key, int fieldNo) to return the appropriate field as a String
public static String getName(String key) {
try {
// Need single quote marks ' around the key field in SQL. This is easy to get wrong!
// For instance if key was "11" the SELECT statement would be:
// SELECT * FROM Stock WHERE stockKey = '11'
ResultSet res = stmt.executeQuery("SELECT * FROM Stock WHERE stockKey = '" + key + "'");
if (res.next()) { // there is a result
// the name field is the second one in the ResultSet
// Note that with ResultSet we count the fields starting from 1
return res.getString(2);
} else {
return null;
}
} catch (SQLException e) {
System.out.println(e);
return null;
}
}
public static double getPrice(String key) {
// Similar to getName. If no result, return -1.0
return 0;
}
public static int getQuantity(String key) {
// Similar to getName. If no result, return -1
return 0;
}
// update stock levels
// extra is +ve if adding stock
// extra is -ve if selling stock
public static void update(String key, int extra) {
// SQL UPDATE statement required. For instance if extra is 5 and stockKey is "11" then updateStr is
// UPDATE Stock SET stockQuantity = stockQuantity + 5 WHERE stockKey = '11'
String updateStr = "UPDATE Stock SET stockQuantity = stockQuantity + " + extra + " WHERE stockKey = '" + key + "'";
System.out.println(updateStr);
try {
stmt.executeUpdate(updateStr);
} catch (SQLException e) {
System.out.println(e);
}
}
// close the database
public static void close() {
try {
connection.close();
} catch (SQLException e) {
// this shouldn't happen
System.out.println(e);
}
}
}
Sorry if this seems a stupid question but I am fairly new to Java and was making good progress until this roadblock.
Thanks in advance!
Alex
Searching for "java sql" on Google delivers this link: https://docs.oracle.com/javase/tutorial/jdbc/basics/processingsqlstatements.html
From a connection you can create a statement (you can find this in the link and in your code) , then fetch a result set and loop over that with rs.next(). That should get your started.
Of course you have to make sure that the driver and database are there/running, just saying...
Here netbeans has nothing to do with database. This is a Java-based integrated development environment(IDE) that will help you to reduce syntactic error.
public void dataAccess(){
try {
String connectionUrl = "suitable connection url as per your database";
Connection con = null;
Statement stmt = null;
ResultSet rs = null;
Class.forName("JDBC driver name as per your database");
con = DriverManager.getConnection(connectionUrl, userName, password);
String SQL = "SQL query as per your criteria";
stmt = con.createStatement();
rs = stmt.executeQuery(query);
while (rs.next()) {
// look into ResultSet api and use method as per your requirement
}
rs.close();
}
catch (Exception e) {
//log error message ;
}
}
In the following code, I execute a query on a SQLite JDBC connection via the executeRestitutionalQuery(String query) method:
public static ArrayList<Metadata> findMetadata(String name, String text, String after, String before, String repPath)
throws SQLException, ClassNotFoundException {
ArrayList<Metadata> data = new ArrayList<Metadata>();
boolean needADD = false;
String query = "SELECT * from " + TABLE_NAME_METADATA;
...
query += " ORDER BY timestamp DESC;";
ResultBundle bundle = executeRestitutionalQuery(query);
ResultSet result = bundle.getResultSet();
while(result.next()){
Metadata metadata = new Metadata(result.getLong("id"), result.getString("name"), Timestamp.valueOf(result.getString("timestamp")),
result.getInt("filesNo"), result.getLong("size"), result.getString("description"), -1);
data.add(metadata);
}
closeStatementAndResultSet(bundle.getStatement(), bundle.getResultSet());
return data;
}
private static ResultBundle executeRestitutionalQuery(String query) throws SQLException, ClassNotFoundException{
Connection connection = null;
Statement statement = null;
ResultSet result = null;
ResultBundle bundle = null;
try{
connection = getConnection();
statement = connection.createStatement();
statement.executeUpdate(query);
connection.commit();
result = statement.executeQuery(query);
bundle = new ResultBundle(statement, result);
}finally{
if(connection != null){
try{
connection.close();
}catch (Exception e){
/* ignored */
}
}
}
return bundle;
}
private static void closeStatementAndResultSet(Statement statement, ResultSet result){
if(result != null){
try{
result.close();
}catch (Exception e){
// ignored
}
}
if(statement != null){
try{
statement.close();
}catch (Exception e){
// ignored
}
}
}
The ResultBundle class is just used to summarize the resultset and the statement. It looks like this:
public class ResultBundle {
private final Statement statement;
private final ResultSet result;
public ResultBundle(Statement statement, ResultSet result){
this.result = result;
this.statement = statement;
}
public Statement getStatement(){
return statement;
}
public ResultSet getResultSet(){
return result;
}
}
The problem is, that every call to result.getLong(), result.getString() etc. returns null resp. 0. I can't understand why. The queries should all be okay, as the code was running fine before I had to do some refactoring. Could the problem arise from the ResultBundle-class? What am I not seeing here?
Statements and ResultSets are "live" objects, living only as long as their connection. The executeRestitutionalQuery returns a ResultBundle, whose result and statement members are implicitly closed on return when the connection is closed in the finally block.
try {
...
}finally{
if(connection != null){
try{
connection.close(); // <---- here's the problem
}catch (Exception e){
/* ignored */
}
}
}
By the time, the caller of executeRestitutionalQuery can lay its hand on the resource bundle, the connection has been closed, and the result set is "dead".
I would say this is a bad design.
A better one would keep the SQL objects in tight scope, map results into a collection or object and immediately close all those scarce resources. Not only will the data be available to clients, but you'll avoid nasty problems with connection and cursors exhausted. It'll scale better, too.
I've got database querying that has become too slow with my current implementation. I need to get all movies from a database and for each of these movies i need their file data from another table. So for each movie i am doing another query. For each candidate entry i need to do a comparison to every movie in the database. Should this be taking 5-10 seconds to execute for approximately 500 candidates?
// get movies with all their versions
private ArrayList<Movie> getDatabaseMovies(Connection conn) throws Exception {
PreparedStatement getMoviesStmt = conn.prepareStatement("SELECT movieid, title FROM movies", Statement.RETURN_GENERATED_KEYS);
ArrayList<Movie> movies = new ArrayList<Movie>();
try {
ResultSet rs = getMoviesStmt.executeQuery();
while (rs.next()) {
Movie movie = new Movie(rs.getString(2), getDatabaseMovieFiles(conn, rs.getInt(1)));
movies.add(movie);
}
} catch (SQLException e) {
e.printStackTrace();
} finally {
getMoviesStmt.close();
}
return movies;
}
public ArrayList<MovieFile> getDatabaseMovieFiles(Connection conn, int movieID) throws Exception {
ArrayList<MovieFile> movieFiles = new ArrayList<MovieFile>();
PreparedStatement stmt = conn.prepareStatement("SELECT filename, size, hash, directory FROM file_video WHERE movieid = ?");
try {
stmt.setInt(1, movieID);
ResultSet rs = stmt.executeQuery();
while (rs.next()) {
MovieFile movieFile = new MovieFile(rs.getString(1), rs.getLong(2), rs.getBytes(3), rs.getString(4));
movieFiles.add(movieFile);
}
} catch (SQLException e) {
e.printStackTrace();
} finally {
stmt.close();
}
return movieFiles;
}
Should this be taking 5-10 seconds to execute for approximately 500 candidates?
Probably not.
There are two ways to improve this:
Make sure that there is an index on the movieid column of file_video.
Combine the two queries into one by using a JOIN.
You probably should do both.