we have a very slow Java API function in SpringBoot framework, and follows is a profiling output of it:
We believe this is coming from this line of code:
The query before that line, retrieves 41,573 rows (given specified accountId and one more parameter) - this is done in acceptable time of about 2 secs.
return (ArrayList) this.pbaJdbc.query(sql, new Object[] { accountId, taxRegId }, bpm);
We tried to set the FetchSize to 1000 (it was set to -1 by default which I was unable to determine the meaning of in documentation) - this change did not improve the performance of the operation noticeably.
One other suggestion was to switch to a HashMap structure instead of ArrayList, but I am not sure how to do that... tried to write a hash map, something like:
HashMap<Integer, accountQueryResult> accmap = new HashMap(Integer, accountQueryResult);
But this errors with Expression expected, I am not sure how to do it right.
This is the relevant piece of code for your review:
#Repository
public class UserRepository {
#Autowired
#Qualifier("pbaJdbc")
private JdbcTemplate pbaJdbc;
public ArrayList<accountQueryResult> getAccountsSubscriptionsResources(int accountId, String taxRegId) {
BeanPropertyRowMapper bpm = new BeanPropertyRowMapper(accountQueryResult.class);
bpm.setPrimitivesDefaultedForNullValue(true);
String sql =
"SELECT " +
"\"Account\".\"AccountID\",\n" +
"\"Account\".\"VendorAccountID\",\n" +
"\"Account\".\"AdminPhAreaCode\",\n" +
"\"Account\".\"AdminPhNumber\",\n" +
"\"Account\".\"AdminFaxAreaCode\",\n" +
"\"Account\".\"AdminFaxNumber\",\n" +
"\"Account\".\"AdminEmail\",\n" +
"\"Account\".\"PersPhAreaCode\",\n" +
"\"Account\".\"PersPhNumber\",\n" +
"\"Account\".\"PersFaxAreaCode\",\n" +
"\"Account\".\"PersFaxNumber\",\n" +
"\"Account\".\"PersEmail\",\n" +
"\"Account\".\"TaxStatus\",\n" +
"\"Account\".\"CompanyName\",\n" +
"\"Account\".\"Address1\",\n" +
"\"Account\".\"Address2\",\n" +
"\"Account\".\"City\",\n" +
"\"Account\".\"Zip\",\n" +
"\"ActivePaytool\".\"CutNumber\",\n" +
"\"ActivePaytool\".\"PaySystem\",\n" +
"\"Subscription\".\"subscriptionID\",\n" +
"\"Subscription\".\"SubscriptionName\",\n" +
"\"Subscription\".\"Status\",\n" +
"\"Subscription\".\"PlanID\",\n" +
"\"PlanPeriod\".\"Period\",\n" +
"\"PlanPeriod\".\"PlanPeriodID\",\n" +
"\"PlanPeriod\".\"PeriodType\",\n" +
"\"PlanPeriod\".\"RenewalFee\",\n" +
"\"PlanPeriod\".\"SetupFee\",\n" +
"\"SubscrParam\".\"resourceID\",\n" +
"\"BMResource\".\"name\" AS \"ResourceName\",\n" +
"\"SubscrParam\".\"IncludedValue\",\n" +
"\"SubscrParam\".\"Amount\",\n" +
"\"SubscrParamValue\".\"IdParameter\",\n" +
"\"SubscrParamValue\".\"Value\",\n" +
"\"IntUsers\".\"UsersID\",\n" +
"\"IntUsers\".\"Login\" AS \"LoginID\"\n" +
"FROM\n" +
"\"Account\"\n" +
"LEFT JOIN \"IntUsers\" ON \"IntUsers\".\"AccountID\" = \"Account\".\"AccountID\"\n" +
"LEFT JOIN \"Subscription\" ON \"Subscription\".\"AccountID\" = \"Account\".\"AccountID\"\n" +
"AND \"Subscription\".\"Status\" IN ('30', '40', '15')\n" +
"LEFT JOIN \"SubscrParam\" ON \"SubscrParam\".\"subscriptionID\" = \"Subscription\".\"subscriptionID\"\n" +
"LEFT JOIN \"BMResource\" ON \"BMResource\".\"resourceID\" = \"SubscrParam\".\"resourceID\"\n" +
"LEFT JOIN \"SubscrParamValue\" ON \"SubscrParamValue\".\"subscriptionID\" = \"Subscription\".\"subscriptionID\"\n" +
"LEFT JOIN \"PlanPeriod\" ON \"PlanPeriod\".\"PlanID\" = \"Subscription\".\"PlanID\"\n" +
"AND NOT (\n" +
"\"Subscription\".\"Period\" = \"PlanPeriod\".\"Period\"\n" +
"AND \"Subscription\".\"PeriodType\" = \"PlanPeriod\".\"PeriodType\"\n" +
")\n" +
"AND \"PlanPeriod\".\"Enabled\" = 1\n" +
"AND \"PlanPeriod\".\"Trial\" = 0\n" +
"LEFT JOIN (\n" +
"SELECT\n" +
" \"DefPayTool\".\"PayToolID\",\n" +
" \"PayTool\".\"CutNumber\",\n" +
" \"PayTool\".\"PaySystem\",\n" +
" \"PayTool\".\"OwnerAccountID\"\n" +
"FROM\n" +
" \"PayTool\"\n" +
"INNER JOIN \"DefPayTool\" ON \"DefPayTool\".\"AccountID\" = \"PayTool\".\"OwnerAccountID\"\n" +
"WHERE\n" +
" \"PayTool\".\"IsSuspended\" = 0\n" +
"GROUP BY\n" +
" \"PayTool\".\"PayToolID\",\n" +
" \"PayTool\".\"CutNumber\",\n" +
" \"PayTool\".\"PaySystem\",\n" +
" \"PayTool\".\"OwnerAccountID\",\n" +
" \"DefPayTool\".\"PayToolID\"\n" +
"HAVING\n" +
" COUNT (*) > 0\n" +
") AS \"ActivePaytool\" ON \"ActivePaytool\".\"OwnerAccountID\" = \"Account\".\"AccountID\"\n" +
"WHERE\n" +
"\"Account\".\"AccountID\" = ? " +
"AND \"Account\".\"TaxRegID\" = ? " +
"ORDER BY\n" +
"\"Account\".\"AccountID\",\n" +
"\"Subscription\".\"subscriptionID\";";
return (ArrayList) this.pbaJdbc.query(sql, new Object[] { accountId, taxRegId }, bpm);
}
Follows is the accountQueryResult class which is currently used as mapper.
package com.store.models.query;
public class accountQueryResult {
private int AccountID;
private int VendorAccountID;
private String CompanyName;
private String AdminPhAreaCode;
private String AdminPhNumber;
private String AdminFaxAreaCode;
private String AdminFaxNumber;
private String AdminEmail;
private String PersPhAreaCode;
private String PersPhNumber;
private String PersFaxAreaCode;
private String PersFaxNumber;
private String PersEmail;
private int TaxStatus;
private String CutNumber;
private String PaySystem;
private int subscriptionID;
private String SubscriptionName;
private int Status;
private int PlanID;
private int resourceID;
private String ResourceName;
private int IncludedValue;
private int Amount;
private String IdParameter;
private String Value;
private int UsersID;
private String LoginID;
private String Address1;
private String Address2;
private String City;
private String Zip;
private int period;
private int planPeriodID;
private int periodType;
private double renewalFee;
private double setupFee;
// Ommited getter and setters
}
Please assist, how to change this one of the suggestions was using JPA, preferably with minimal changes as to avoid having to change all the business logic that processes the resulting array later..
Thanks!
The first thing when tackling a performance question like this is to find out where the time is spent.
Given your profilers information, it strikes me as odd that getting the ints seems to take so much longer than getting the other fields. I see following possible reasons:
the getInt does some expensive conversion, e.g. the JDDBC driver might return BigDecimal which then have to get converted to int.
it is just the first column to get accessed and actually triggers fetching and spends lot's of time waiting.
For the overall performance another problem might be the use of reflection by BeanPropertyRowMapper but this doesn't even show up in the profiler so far.
What I consider more likely is that the statement might return after 2 seconds but actually takes much longer to fetch all the results.
To clarify the situation I'd recommend testing the following:
Inspect the ResultSet returned by the JDBC driver for the datatypes it actually contains and check if you can get something more suitable out of it by changing column types or by casting in the SQL statement.
Obtain all the columns from all the rows without doing anything expensive with them. Just make sure the optimizer doesn't get rid of the access. For example you might want to create a hash from each value and add all hashes and print that at the end. Check the performance of that.
Related
This question already has an answer here:
android.database.sqlite.SQLiteException: no such column or name
(1 answer)
Closed 3 months ago.
I am stuck on an "SQLiteException: no such column:" error. I am trying to make some code to check if an item exists in the database before storing it, I don't know if it's the best way or not but it does the job. Or not really, when I use all numbers for the data in the particular column that I'm searching it works fine. But if there is any letter in the column data it crashes.
The MainActivity
private ConversationsDatabaseHelper db;
//onCreate stuff here
db = new ConversationsDatabaseHelper(this);
String threadId = "886";
Log.d(TAG, "dbTest: EXISTS; " + db.conversationExists(threadId));
and in the databaseHelper class is the converstionExists function
public boolean conversationExists(String threadId) {
// Select All Query
String selectQuery = "SELECT * FROM " + Conversations.TABLE_NAME + " WHERE " +
Conversations.COLUMN_THREAD_ID + " LIKE " + threadId;
SQLiteDatabase db = this.getWritableDatabase();
Cursor cursor = db.rawQuery(selectQuery, null);
boolean returnValue = false;
// Check if this message exists
if (cursor.moveToFirst()) {
returnValue = true;
}
// close db connection
db.close();
return returnValue;
}
So if I use for example "886" as the threadId value then all is fine. If I create a row with matching threadId then it returns true. In this case I did not so hence false.
stack trace...
dbTest: EXISTS; false
but using a886 results in
Caused by: android.database.sqlite.SQLiteException: no such column: a886 (code 1 SQLITE_ERROR): , while compiling: SELECT * FROM conversations WHERE thread_id LIKE a886
and 88a6 results in this
Caused by: android.database.sqlite.SQLiteException: unrecognized token: "88a6" (code 1 SQLITE_ERROR): , while compiling: SELECT * FROM conversations WHERE thread_id LIKE 88a6
It almost looks like mixing letters and numbers might be part of the reason here but should not be as the column was created to hold TEXT datatype. here is the database create table query string.
public static final String TABLE_NAME = "conversations";
public static final String COLUMN_ID = "id";
public static final String COLUMN__ID = "_id";
public static final String COLUMN_GROUP_ID = "group_id";
public static final String COLUMN_LAST_MESSAGE_ID = "last_message_id";
public static final String COLUMN_THREAD_ID = "thread_id";
public static final String COLUMN_ADDRESS = "address";
public static final String COLUMN_CONTACT = "contact";
public static final String COLUMN_BODY = "body";
public static final String COLUMN_DATE = "date";
public static final String COLUMN_TYPE = "type";
public static final String COLUMN_STATE = "state";
public static final String COLUMN_READ = "read";
public static final String COLUMN_STATUS = "status";
public static final String COLUMN_CT = "ct";
// Create table SQL query
public static final String CREATE_TABLE =
"CREATE TABLE " + TABLE_NAME + "("
+ COLUMN_ID + " INTEGER PRIMARY KEY ,"
+ COLUMN_LAST_MESSAGE_ID + " TEXT,"
+ COLUMN__ID + " TEXT,"
+ COLUMN_GROUP_ID + " TEXT,"
+ COLUMN_THREAD_ID + " TEXT,"
+ COLUMN_ADDRESS + " TEXT,"
+ COLUMN_CONTACT + " TEXT,"
+ COLUMN_BODY + " TEXT,"
+ COLUMN_DATE + " TEXT,"
+ COLUMN_TYPE + " TEXT,"
+ COLUMN_STATE + " TEXT,"
+ COLUMN_READ + " TEXT,"
+ COLUMN_STATUS + " TEXT,"
+ COLUMN_CT + " TEXT"
+ ")";
I am "up a creek" with this and any help would be greatly appreciated.
The value a886 should be a text/string literal not a numeric literal. Therefore it should be enclosed in single quotes. The errors are because:-
a886 fails with no column found as it's taken to be a column name as it's not a literal.
whilst 88a6 is first not a valid literal (due to the a) and therefore a column name but then an invalid column (cannot start with a numeric unless enclosed) name and thus not a known token.
See Literal Values (Constants)
You could fix this using (enclosing the threadId in single quotes ) :-
String selectQuery = "SELECT * FROM " + Conversations.TABLE_NAME + " WHERE " +
Conversations.COLUMN_THREAD_ID + " LIKE '" + threadId + "'";
However, it is recommended to use bound parameters to protect against SQL Injection.
Thus it would be recommended to use :-
String selectQuery = "SELECT * FROM " + Conversations.TABLE_NAME + " WHERE " +
Conversations.COLUMN_THREAD_ID + " LIKE ?";
along with :-
Cursor cursor = db.rawQuery(selectQuery, new String[]{threadId});
i.e. the ? is replaced by the threadId value properly enclosed.
You may wish to consider using the convenience query method rather than rawQuery, this would be :-
Cursor cursor = db.query(Conversations.TABLE_NAME,null,Conversations.COLUMN_THREAD_ID + " LIKE ?", new String[]{theadId},null,null, null);
The SQL is built for you.
It almost looks like mixing letters and numbers might be part of the reason here but should not be as the column was created to hold TEXT datatype.
SQlite has no issue storing any value in any type. The type, which itself can be virtually anything (rules are used to determine the resultant type), is only an indication of the value that will be stored. The only exception is that a rowid or an alias of the rowid MUST be an integer value (your id column is an alias of the rowid column).
i use java from a lot of time but i never found a better "graphical" way for concat strings, when i write SQL query i have a hard to read query for all the concats i made (i use a java class for the name values of all the columns of the db) something like
public static class DBMetaData {
static class LISTINO_TABLE {
static final String TABLE_NAME = "listino";
static final String ID = "_ID";
static final String PRODUCT_NAME_KEY = "nome_prodotto";
static final String CODE_KEY = "codice";
static final String PRICE_KEY = "prezzo";
static final String ENABLE_KEY = "enable";
static final String PREZZO_VENDITA_KEY = "prezzo_vendita";
static final String RICAVO_KEY = "ricavo";
static final String NUMERO_COLLI_KEY = "num_colli";
static final String TIPO_COLLO_KEY = "tipo_collo";
static final String DATA_INSERIMENTO_KEY = "data_inserimento";
}
and when i write the query i have query like that
String sql = "select *,lo." + DBMetaData.LISTINO_ORDINE_TABLE.QUANTITY_KEY + " as quant from " + DBMetaData.LISTINO_ORDINE_TABLE.TABLE_NAME +
" lo left join "+DBMetaData.OFFERTE_TABLE.TABLE_NAME + " o on lo."+ DBMetaData.LISTINO_ORDINE_TABLE.CODICE_ID + " = o." + DBMetaData.OFFERTE_TABLE.CODICE_ARTICOLO_KEY +
" left join "+DBMetaData.LISTINO_TABLE.TABLE_NAME + " l on lo."+DBMetaData.LISTINO_ORDINE_TABLE.CODICE_ID + "= l."+DBMetaData.LISTINO_TABLE.CODE_KEY +
" left join " + DBMetaData.INVENTARIO_TABLE.TABLE_NAME + " i on lo."+DBMetaData.LISTINO_ORDINE_TABLE.CODICE_ID + "= i."+DBMetaData.INVENTARIO_TABLE.CODE_KEY +
" where "+DBMetaData.LISTINO_ORDINE_TABLE.NUMBER_KEY + " = 0 order by "+DBMetaData.LISTINO_ORDINE_TABLE.QUANTITY_KEY + " DESC";
there is a better way for write the query? i know i can just write the query without use my DBMetaData, but if i wanna edit in the future something i have to re-edit all my sqls and the use of the class is a practice i learnt when i started with android and i'm using it but i don't like it :-)
any tips? tnx
You could use String.format to remove the need for all the string concatenation.
For example:
String sql = String.format("select * from %s", TABLE_NAME);
You could also use Apache Commons Text's StringSubstitutor.
final Map<String, String> valuesMap = new HashMap<>();
valuesMap.put("tableName", "Table");
//etc...
final String string = "select * from ${tableName}";
final String sql = new StringSubstitutor(valuesMap).replace(string);
//"select * from Table"
I have some string const inside code. Can I put all this into resources strings.xml? I assume that the possibility of separate use of column names will remain. It is necessary to search for the index of posts in future requests to the database. Not to use the indexes the numbers is confusing, uncomfortable.
private static final String NAME_DB = "cars.db";
private static final String TABLE_NAME = "cars";
private static final String COL_BRAND = "brand";
private static final String COL_MODEL = "model";
private static final String COL_COLOR = "color";
private static final String COL_MAX_SPEED = "max_speed";
private static final String COL_ENGINE_POWER = "engine_power";
private static final String CREATE_TABLE =
"CREATE TABLE IF NOT EXISTS " + TABLE_NAME + " (" +
"id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL," +
COL_BRAND + " TEXT NOT NULL, " +
COL_MODEL + " TEXT NOT NULL, " +
COL_COLOR + " TEXT, " +
COL_MAX_SPEED + " INTEGER," +
COL_ENGINE_POWER + " INTEGER)";
private static final String INSERT_DEFAULT_VALUES =
"INSERT INTO " + TABLE_NAME + "(brand, model, color, max_speed, engine_power) VALUES" +
"('Audi', 'Q7 II', 'White', 233, 252)," +
"('Acura', 'ZDX', 'Black', 241, 300)," +
"('BMW', 'X1 I', 'Blue', 205, 204)," +
"('Hyundai', 'Solaris', 'Red', 100, 100)," +
"('Hyundai', 'Tucson', 'Gray', 150, 177)," +
"('BMW', '8 G14', 'Black', 250, 320)";
private static final String SELECT_ALL = "SELECT * FROM " + TABLE_NAME;
private static final String SELECT_TOP_1 = "SELECT * FROM " + TABLE_NAME + " LIMIT 1";
Example code
DatabaseAdapter(Context context) {
db = context.openOrCreateDatabase(NAME_DB, MODE_PRIVATE, null);
db.execSQL(CREATE_TABLE);
Cursor cursor = db.rawQuery(SELECT_TOP_1, null);
if (!cursor.moveToFirst()) {
db.execSQL(INSERT_DEFAULT_VALUES);
}
}
String[] selectAll() {
Cursor cursor = db.rawQuery(SELECT_ALL, null);
if (!cursor.moveToFirst()) {
return null;
}
ArrayList<String> list = new ArrayList<>();
while (cursor.moveToNext()) {
list.add(
cursor.getString(cursor.getColumnIndex(COL_BRAND)) + " " +
cursor.getString(cursor.getColumnIndex(COL_MODEL)) + " " +
cursor.getString(cursor.getColumnIndex(COL_COLOR)) + " " +
cursor.getString(cursor.getColumnIndex(COL_MAX_SPEED)) + " " +
cursor.getString(cursor.getColumnIndex(COL_ENGINE_POWER))
);
}
return list.toArray(new String[0]);
}
There are some solutions with SQL in XML. Look if you can use one of those.
However IMHO it is better to centralize the code and keep text there, without constants even.
Separate concerns: database query and resulting object. This can be done by providing a Stream of a low-level result set object, that using classes can map to their own classes.
Decouples classes; keeps the layers separate. (Maybe less interesting on Android.)
The using code becomes nicely compact, the queries being elsewhere.
My arguments:
Less back and forth jumping to different sources
Better readability
Column names do not change
Capitals of constants are less readable
Keeping the code together prevents similar queries to be spread around unattended
However it still is better than SQL in the higher level business code, to isolate SQL in XML to have smaller source, queries elsewhere. So at least do that.
Java - Storing SQL statements in an external file
I need to pull few fields from entity class Employee and add few extra hard coded field and return the result using GROUP BY clause.
Below is the code I tried:
String query = "SELECT emp.category, emp.salary 0 as somevalue, 0 as dummy FROM employee emp "
+ "WHERE emp.date = :date AND emp.class = :class AND emp.classificationDetail.shortDescription = :classificationType GROUP BY emp.category";
TypedQuery<CustomEmployee> typQuery = entityManager.createQuery(query, CustomEmployee.class);
typQuery.setParameter("date", req.getDate());
typQuery.setParameter("class", req.getClass());
return typQuery.getResultList();
But I am getting exception that Cannot create TypedQuery for query with more than one return using requested result type.
How to achieve this.
Thanks.
First check this part: emp.salary 0 as somevalue. This should be either emp.salary as somevalue or 0 as somevalue, but not both.
Define a class like following (to keep it short; I use public properties, but you can change it if you want):
public class CustomEmployee {
public String category;
public Double salary;
public Double dummy;
...
}
The use it in the query as follows:
String query = "SELECT new mypackage.CategorySalary( " +
" emp.category, " +
" emp.salary as somevalue, " +
" 0 as dummy " +
") from ... " +
"WHERE ... ";
TypedQuery<CustomEmployee> typQuery = entityManager.createQuery(query, CustomEmployee.class);
public static final String UPDATE_DOCUMENTS_WITH_TO_DELETE_FLAG_FOR_USER_SQL = "\n" +
"UPDATE document d \n" +
"SET d.indexed = :flagValue \n" +
"WHERE d.user_id = :userId \n" +
"AND d.to_delete = :toDelete";
public static final String UPDATE_DOCUMENTS_WITH_TO_DELETE_FLAG_FOR_USER_WITH_EXCEPTIONS_SQL = "\n" +
"UPDATE document d \n" +
"SET d.indexed = :flagValue \n" +
"WHERE d.user_id = :userId \n" +
"AND d.to_delete = :toDelete \n" +
"AND d.id NOT IN (:exceptForDocuments)";
public int markUserDocumentsToDeleteAsUnindexed(String userId,Collection<String> exceptForDocuments) {
Map<String,Object> params = Maps.newHashMap();
params.put("flagValue",false);
params.put("userId",userId);
params.put("toDelete",1);
params.put("exceptForDocuments",exceptForDocuments);
if ( exceptForDocuments.isEmpty() ) {
return jdbcTemplate.update(UPDATE_DOCUMENTS_WITH_TO_DELETE_FLAG_FOR_USER_SQL, params);
}
else {
return jdbcTemplate.update(UPDATE_DOCUMENTS_WITH_TO_DELETE_FLAG_FOR_USER_WITH_EXCEPTIONS_SQL,params);
}
}
Is there a way to use a single query to perform both updates?
Because actually using the UPDATE_DOCUMENTS_WITH_TO_DELETE_FLAG_FOR_USER_WITH_EXCEPTIONS_SQL query seems to work against H2, but not MySQL.
Any idea to avoid this query duplication?
The problem is likely because not every driver can handle parameterized arrays/collections. If you have complete control over the exceptForDocuments contents, you can serialize it to SQL yourself (with simple sanitization checks) and then conditionally append it without using parameters.