We Keep Coding

Memory leak Java, HEAP size not reducing

So I grow up with C++ and have no problem with memory management whatsoever building C++ programs. But the time has come the market is asking me to understand and learn Java. Okay no problem! After spending a lot of time developing Java applications I keep hitting my head with this memory management in Java.
I took one of my applications and trying to understand how to keep a low heap size. The image bellow is a print-screen from VisualVM 2.0.1. The numbers represent the times I call the CollectData() function. As you can see, call number 1 is all okay, but when I call it again it grows with 125MB and keeps going on. I don't understand why? (So I paste my code bellow, so maybe someone could help me explaining whats going on).
So I tried basically everything with playing with the null and new operators, without any success. Also I tried playing with the System.gc() function.
public void CollectData() {
/// Variables
taBB.setText("");
Connection con;
String tb_name_hours = "table_name";
String tb_name_minutes = "table_name";
String db_name = "database_name";
List<TickerIntervalData> data_set_mins = new ArrayList<TickerIntervalData>();
List<TickerIntervalData> data_set_hours = new ArrayList<TickerIntervalData>();
DecimalFormat dec = new DecimalFormat("#0.0000");
DecimalFormat dec2 = new DecimalFormat("#0.00000000");
///
try {
/// Connect
String url = "jdbc:mysql://" + tfIP.getText() + ":" + tfPort.getText();
Class.forName("com.mysql.cj.jdbc.Driver").newInstance();
DriverManager.setLoginTimeout(2);
con = DriverManager.getConnection(url, tfUser.getText(), String.valueOf(tfPassword.getPassword()));
// Use db_name
Statement stt = con.createStatement();
stt.execute("USE " + db_name);
// Get rows from hour table
ResultSet res = stt.executeQuery("SELECT * FROM " + tb_name_hours);
while (res.next()) {
TickerIntervalData new_record = new TickerIntervalData();
new_record.setOpen(res.getDouble("open"));
new_record.setHighest(res.getDouble("highest"));
new_record.setLowest(res.getDouble("lowest"));
new_record.setClose(res.getDouble("close"));
new_record.setVolume(res.getDouble("volume"));
new_record.setBv(res.getDouble("bv"));
new_record.setTimestamp(res.getString("time"));
data_set_hours.add(new_record);
}
// Get rows from minute table
res = stt.executeQuery("SELECT * FROM " + tb_name_minutes);
while (res.next()) {
TickerIntervalData new_record = new TickerIntervalData();
new_record.setOpen(res.getDouble("open"));
new_record.setHighest(res.getDouble("highest"));
new_record.setLowest(res.getDouble("lowest"));
new_record.setClose(res.getDouble("close"));
new_record.setVolume(res.getDouble("volume"));
new_record.setBv(res.getDouble("bv"));
new_record.setTimestamp(res.getString("time"));
data_set_mins.add(new_record);
}
// Free resources
res.close();
stt.close();
res = null;
stt = null;
//
data_set_hours.clear();
data_set_hours = null;
data_set_mins.clear();
data_set_mins = null;
} catch (Exception e) {
taBB.setText(e.toString());
}
}

using JPA, method is taking a 60 sec

I am working with JPA, my web application is taking 60 sec to execute this method, I want to execute it faster how to achive ?
public boolean evaluateStudentTestPaper (long testPostID, long studentID, long howManyTimeWroteExam) {
Gson uday = new Gson();
Logger custLogger = Logger.getLogger("StudentDao.java");
// custLogger.info("evaluateTestPaper test paper for testPostID: " +
// testPostID);
long subjectID = 0;
// checking in table
EntityManagerFactory EMF = EntityManagerFactoryProvider.get();
EntityManager em = EMF.createEntityManager();
List<StudentExamResponse> studentExamResponses = null;
try {
studentExamResponses = em
.createQuery(
"SELECT o FROM StudentExamResponse o where o.studentId=:studentId And o.testPostID=:testPostID and o.howManyTimeWroteExam=:howManyTimeWroteExam")
.setParameter("studentId", studentID).setParameter("testPostID", testPostID)
.setParameter("howManyTimeWroteExam", howManyTimeWroteExam).getResultList();
System.out.println("studentExamResponses--------------------------------------------------"
+ uday.toJson(studentExamResponses) + "---------------------------------------");
} catch (Exception e) {
custLogger.info("exception at getting student details:" + e.toString());
studentExamResponses = null;
}
int studentExamResponseSize = studentExamResponses.size();
if (AppConstants.SHOWLOGS.equalsIgnoreCase("true")) {
custLogger.info("student questions list:" + studentExamResponseSize);
}
// Get all questions based on student id and test post id
List<ExamPaperRequest> examPaperRequestList = new ArrayList<ExamPaperRequest>();
List<Questions> questionsList = new ArrayList<Questions>();
// StudentExamResponse [] studentExamResponsesArgs =
// (StudentExamResponse[]) studentExamResponses.toArray();
// custLogger.info("Total questions to be evaluated: " +
// examPaperRequestList.size());
List<StudentTestResults> studentTestResultsList = new ArrayList<StudentTestResults>();
StudentTestResults studentTestResults = null;
StudentResults studentResults = null;
String subjectnames = "", subjectMarks = "";
int count = 0;
boolean lastIndex = false;
if (studentExamResponses != null && studentExamResponseSize > 0) {
// studentExamResponses.forEach(studentExamResponses->{
for (StudentExamResponse o : studentExamResponses.stream().parallel()) {
// 900 lines of coade inside which includes getting data from database Queries
}
}

As #Nikos Paraskevopoulos mentioned, it should probably be the ~900 * N database iterations inside that for loop.
I'd say to avoid DB iterations as much as you can, specially inside a loop like that.
You can try to elaborate your current StudentExamResponse sql to englobe more clauses - those you're using inside your for mainly, which could even diminish the amount of items you iterate upon.

My guess would be your select query is taking time.
If possible, set query timeout to less than 60 seconds & confirm this.
Ways of setting query timeout can be found out there - How to set the timeout period on a JPA EntityManager query
If this is because of query, then you may need to work to make select query optimal.

Getting values from multi-dimensional array too slow

Good Day,
I've been working on an update in an android project, and came across an issue. I have to read questions from an SQLite database which i've done successfully by loading it into a multi-dimensional array as shown below in my database helper class:
public String getSome(int s,int t, String Table_Name){
String selectQuery = "SELECT * FROM " + Table_Name;
SQLiteDatabase db = this.getReadableDatabase();
Cursor cursor = db.rawQuery(selectQuery, null);
int rows = cursor.getCount();
int num=0;
int col = 0;
String[][] base = new String[rows][13];
if (cursor.moveToFirst()) {
do {
for (col=0;col<13;++col ){
base[num][col] = (cursor.getString(col));}
++num;
} while (cursor.moveToNext());
return base[s][t];
}
return null;
}
With that done, i read the questions as such in my question class:
public void database_calls(){
setCourseTag(courseTag);
myDbHelper = new DataBaseHelper(getActivity());
try {
myDbHelper.createDataBase();
} catch (IOException ioe) {
throw new Error("Unable to create database");
}
try {
myDbHelper.openDataBase();
}catch(SQLException sqle){
throw sqle;
}
String no= myDbHelper.getSome(ques,0, getCourseTag());
String qu = myDbHelper.getSome(ques,1, getCourseTag());
String a = myDbHelper.getSome(ques,2, getCourseTag());
String b = myDbHelper.getSome(ques,3, getCourseTag());
String c = myDbHelper.getSome(ques,4, getCourseTag());
String d = myDbHelper.getSome(ques,5, getCourseTag());
ans = myDbHelper.getSome(ques,6, getCourseTag());
img = Integer.parseInt(myDbHelper.getSome(ques,8, getCourseTag()));
exp = myDbHelper.getSome(ques,9, getCourseTag());
year = myDbHelper.getSome(ques,10, getCourseTag());
questionImage = myDbHelper.getSome(ques,11, getCourseTag());
length = myDbHelper.getMax(getCourseTag());
}
So recently, i tried to use the year column, (i.e, column 10) to qualify the questions chosen for each quiz session, so that the user may be able to select the questions from any year, he/she wants to attempt. In order to do this, i used a loop at the beginning of the activity to filter out only the required year past questions. Then i transferred the indices of each question to a set, bal , from where it is iterated and so on..
public void countYearQuestions(){
for(int y = 0; y < length; ++y){
//year = myDbHelper.getSome(y,10, getCourseTag());
if (selectedYear.equals(myDbHelper.getSome(y,10, getCourseTag())))
bal.add(y);
}
}
Here, length is the size of the entire question database, for the course, (indicated by getCourseTag()). The code works quite alright. But it takes a whole 8-9secs!! for the activity to load. Any help on how to reduce this loading time would be appreciated.

The way you're doing this is pretty efficient in slowing everything down (and your helper helps you with it a lot):
String no= myDbHelper.getSome(ques,0, getCourseTag());
In each such line you execute a query, create a 2D array holding the whole table and throw nearly everything away. And you fail to close the Cursor.
So you need 12 values from a table and read instead the whole table 12 times.

Improve the search functionality

How can I improve the search functionality.? I have written some codes to search for something.The search was taking too much time. And the code snippets here,
I am pulling the data from the database using this method.,
OracleConnection connection = null;
OraclePreparedStatement ptmst = null;
OracleResultSet rs = null;
OracleCallableStatement cstmt = null;
StringBuffer strBfr = new StringBuffer();
ArrayList myList = new ArrayList();
try
{
connection = (OracleConnection) TransactionScope.getConnection();
strBfr.append("select distinct .......... ");
ptmst = (OraclePreparedStatement)connection.prepareStatement(strBfr.toString());
rs = (OracleResultSet)ptmst.executeQuery();
while (rs.next())
{
HashMap hashItems = new HashMap();
hashItems.put("first",rs.getString(1));
hashItems.put("second",rs.getString(2));
myList.add(hashItems);
}
}
catch (Exception e) {
}
finally {
try {
if (ptmst != null) {
ptmst.close();
}
} catch (Exception e) {
}
try {
if (connection != null) {
TransactionScope.releaseConnection(connection);
}
} catch (Exception e) {
}
}
return myList;
In my jsp:
ArrayList getValues = new ArrayList();
getValues = //calling Method here.
for(int i=0; i < getValues.size();i++)
{
HashMap quoteSrch=(HashMap)allPOV.get(i);
first = (String)quoteSrch.get("first");
second = (String)quoteSrch.get("second");
}
Query:
SELECT DISTINCT(mtl.segment1),
mtl.description ,
mtl.inventory_item_id ,
mtl.attribute16
FROM mtl_system_items_b mtl,
mtl_system_items_tl k
WHERE 1 =1
AND mtl.organization_id = ?
AND k.inventory_item_id = mtl.inventory_item_id
AND NVL(orderable_on_web_flag,'N')= 'Y'
AND NVL(web_status,'UNPUBLISHED') = 'PUBLISHED'
AND mtl.SEGMENT1 LIKE ? --Here is the search term

Make sure organization_id , inventory_item_id and especially SEGMENT1 is indexed in your table.
Your query is pretty standard , if that doesn't work then it seems like your DB server is responding slow which could be due to number of reasons like low space , low memory , slow disk/read etc.
You can then ask your DBA/Server admins to check that.

First you need to find out the real problem
Is it the DB query
Is it the Network (is the App and the DB located on the same machine?)
Once you have identified that it is the DB query, then it becomes more of a DB question.
How does the two tables look like?
Any index used?
How does the data look like (How many rows etc)
After you have analyzed this, you should be able to post the question differently and expect an answer. I am not a DB guy, but I am sure someone would be able to provide some pointers.

Tunning has to be done:
Check TransactionScope.getConnection(); is giving connection without any delay.
Instead of creating new HashMap hashItems = new HashMap(); you can use
while (rs.next()){
myList.add(rs.getString(1) + "delimiter" + rs.getString(2));
}
in jsp use
first = allPOV.get(i).split("delimter")[0];
second = allPOV.get(i).split("delimter")[1];
so that you can reduce memory.
If possible use limit in your query, and use index on SEGMENT1 link.

Why does the NetBeans Java debugger never reach this code?

I'm trying to debug a method in Java using NetBeans.
That method is:
public Integer getNumberOfClamps(Locations paLocation) {
Integer ret = -1;
List list = new ArrayList();
String removeme = "ABC";
if (paLocation == null) {
return ret;
}
try {
IO io = new IO(this.getSchemaName());
Session session = io.getSession();
String sql = "select count(*) from assets a join assettypes at on (at.id = a.assettype_id) ";
sql += "where a.currentlocation_id = " + paLocation.getId() + " and at.clamp = 1 and at.active = 1;";
list = session.createQuery(sql).list();
// for some reason, list is empty yet MySQL reports 40 records
// and the following two lines are never reached!
ret = list.size();
removeme = "WHAT???";
} catch (Exception ex) {
ret = -1; // explicitly set return
} finally {
return ret;
}
}
Towards the middle of the method you will see list = session.createQuery(sql).list();
For some reason, this is returning an empty list even though when the SQL is run manually, I get 40 results.
But the odd part is that once the .list() is called, it jumps to the finally block and never reaches the rest! So for testing, 'removeme' should equal WHAT??? but the debugger reports it as still ABC.
What gives?

You are using the wrong method. 'createQuery' is expecting HQL syntax. Change your method to 'createSQLQuery'