I am trying to create an ordering application in Java where the user places an order, specifying the products. The combination of the order's timestamp and customer's id is unique. What I'm trying to do is to check for new orders every few seconds(or milliseconds), fetch the new orders and then create the corresponding objects. I have an ArrayList of orders in which, each Order has an ArrayList of Products, a timestamp and a customer id.
I send queries to the DB every 700ms in order to retrieve the new orders and create the Objects.
My problem is that sometimes, the query doesn't return the new order, that is, the new entries in the DB. I tried various values for the milliseconds but if it's to small, the program creates two or even more Order objects. I want to exactly one order Object for every new entry.
Here's the code:
DBConnect db = new DBConnect();
ScheduledExecutorService executorService =
Executors.newSingleThreadScheduledExecutor();
String today = new SimpleDateFormat("YYYY-MM-dd").format(new Date());
today = today + " 00:00:00";
ArrayList<Order> orders = new ArrayList<Order>();
db.connect();
executorService.scheduleAtFixedRate(new Runnable()
{
#Override
public void run()
{
String now = new SimpleDateFormat("YYYY-MM-dd HH:mm:ss").format(new Date());
try
{
String query = "SELECT DISTINCT last_edit, customer_id, employee_username, closed\r\n" +
"FROM orders\r\n" +
"WHERE last_edit = '" + now + "' ORDER BY last_edit";
ResultSet rs = db.getStatement().executeQuery(query);
while(rs.next())
orders.add(new Order(rs.getString("last_edit"), rs.getString("customer_id"), rs.getString("employee_username"), rs.getString("closed")));
}
catch(Exception ex)
{
System.out.println(ex);
}
try
{
String query = "SELECT last_edit, product.id AS product_id,\r\n" +
"quantity_weight,\r\n" +
"orders.price\r\n" +
"FROM customer, product, orders\r\n" +
"WHERE orders.customer_id = customer.id\r\n" +
"AND orders.product_id = product.id\r\n" +
"AND last_edit = '" + now + "' ORDER BY last_edit";
ResultSet rs = db.getStatement().executeQuery(query);
for(Order order: orders)
{
while(rs.next())
if(order.getLast_edit().equals(rs.getString("last_edit")))
{
order.getProducts().add(new Product(rs.getString("product_id"), rs.getString("quantity_weight"), rs.getString("orders.price")));
System.out.println(order.getLast_edit());
}
rs.beforeFirst();
}
}
catch (Exception ex)
{
System.out.println(ex);
}
}
}, 0, 700, TimeUnit.MILLISECONDS);
Why are you using equality = insteed of >= condition is a mistery to me. Besides that you relay on system timestamp insteed of timestamp of last newest order. In edge cases multiple orders can have the same timestamp and this could get you it a trouble. I assume that you ahve autoincrement primary key set in your table, so you should rely on id (your pk), that is guaranteed to be unique, insteed of timestamp. Your process would look like this:
Fetch list of all orders - process them and take max PK from selected orders
SELECT (whatever) FROM (somewhere) WHERE order.id > lastSavedId
Process new orders, save max id and wait whatever you want
Go to 2. repeat
Related
I am using the below lines of code to fetch the data from Google Bigquery.
public class BQTEST {
public static void main(String... args) throws Exception {
String datasetName = "mydataset";
String tableName = "mytable";
String projectId = "id-gcs";
String query =
"SELECT id, " +
"qtr, " +
"sales, " +
"year " +
"FROM `id-gcs.mydataset.mytable` " +
"where MOD(id,2) = 0";
BigQuery bigquery = BigQueryOptions.newBuilder().setProjectId(projectId)
.setCredentials(
ServiceAccountCredentials.fromStream(new
FileInputStream("gcs.json"))
)
.build().getService();
TableId tableId = TableId.of(projectId, datasetName, tableName);
QueryJobConfiguration queryConfig = QueryJobConfiguration
.newBuilder(query)
.setPriority(QueryJobConfiguration.Priority.BATCH)
.build();
try {
bigquery.query(queryConfig);
} catch (InterruptedException e) {
e.printStackTrace();
throw new RuntimeException(e.getMessage());
}
TableResult results = bigquery.listTableData(
tableId,
BigQuery.TableDataListOption.pageSize(1)
);
for (FieldValueList row : results.iterateAll()) {
System.out.printf(
"ID: %s qtr: %s sales: %s year: %s\n", row.get(0).getValue(), row.get(1).getValue(), row.get(2).getValue(), row.get(3).getValue());
}
}
}
I have 12 records in a source table starting id value from 1,2,3...12. Since I applied Mod on ID the result set should be with the id value as 2,4,6,8,10,12.
Instead, it's returning the whole data as a resultset.
So, the condition in where clause is not applied.
Seeking help on this .
You're doing two unrelated things here: running a query, and then attempting to read the rows directly from the source table you just queried. The reason you're not getting filtered results is you're not reading rows from the query results, you're reading rows directly from the source table.
Upon further review, it looks like this is based on some sample code that's misleading; I'll get that addressed.
A short example that may be more illuminating: https://cloud.google.com/bigquery/docs/samples/bigquery-query#bigquery_query-java
Notable, see how the result iterator is returned from bigquery.query(). That's where your filtered results are available, not by iterating against the source table.
I have a table of products containing an item's product number and other details, and a ReviewTable with the product number, rating and review. Because an item can have multiple ratings and reviews I need to retrieve all ratings and reviews for that item.
"drop table ProductTable", // The product table
"create table ProductTable ("+
"productNo Char(4)," + // Unique product number
"description Varchar(40)," +
"picture Varchar(80)," +
"price Float)",
"insert into ProductTable values " +
"('0001', '40 inch LED HD TV', 'images/pic0001.jpg', 269.00)",
"drop table ReviewTable",
"create table ReviewTable ("+
"productNo Char(4)," +
"ratingScore Integer," +
"review Varchar(200))",
"insert into ReviewTable values ( '0001', 2, 'Very high quality, but I had to replace it after 1 year' )",
"insert into ReviewTable values ( '0001', 3, 'Very good' )", // Two reviews for the same product
"select * from ReviewTable, ProductTable " +
" where ReviewTable.productNo = ProductTable.productNo",
I have a Product object that takes as arguments the product number, an array of all its ratings and an array of its reviews:
public Product(String aProductNum, double[] aRating, String[] aReviews) {
theProductNum = aProductNum; // Product number
theRating = aRating; // ratings of product
theReviews = aReviews; // All the reviews for the product
}
Finally, I have a getDetails method that retrieves the data about a product, and this is where I need to add multiple values to an array..
public synchronized Product getDetails( String pNum )
throws StockException
{
try
{
String [] reviews = new String[0]; // Initialise reviews
double [] ratings = new double[0]; // Initialise ratings
Product dt = new Product( "0", ratings, reviews); // Initialise product
ResultSet rs = getStatementObject().executeQuery(
"select ratingScore, review " +
" from ProductTable, ReviewTable " +
" where ProductTable.productNo = '" + pNum + "' " +
" and ReviewTable.productNo = '" + pNum + "'"
);
if ( rs.next() )
{
dt.setProductNum( pNum );
dt.setRating(rs.getDouble("ratingScore") ); // CURRENTLY PULLING ONLY ONE RATING
dt.setReviews(rs.getString("review")); // CURRENTLY PULLING ONLY ONE REVIEW
}
rs.close();
return dt;
} catch ( SQLException e )
{
throw new StockException( "SQL getDetails: " + e.getMessage() );
}
}
Any help please? Thanks a lot in advance
This is a rewrite of method getDetails. Explanations after the code.
Note that this is based entirely and solely on the details in your question.
public synchronized Product getDetails( String pNum )
throws StockException
{
String sql = "select ratingScore, review from ReviewTable where productNo = ?";
try (java.sql.Connection conn = java.sql.DriverManager.getConnection("URL");
java.sql.PreparedStatement ps = conn.prepareStatement(sql))
{
ps.setString(1, pNum);
ResultSet rs = ps.executeQuery();
java.util.List<Double> ratings = new java.util.ArrayList<>();
java.util.List<String> reviews = new java.util.ArrayList<>();
while ( rs.next() )
{
ratings.add(rs.getDouble(1);
reviews.add(rs.getString(2));
}
double[] ratingsArray = ratings.stream().mapToDouble(Double::doubleValue).toArray();
String[] reviewsArray = reviews.toArray(new String[]{});
return new Product(pNum, ratingsArray, reviewsArray);
} catch ( SQLException e )
{
throw new StockException( "SQL getDetails: " + e.getMessage() );
}
}
You should use java.sql.PreparedStatement rather than Statement.
The above code uses try-with-resources since you need to close the PreparedStatement after you finish using it.
Since you only retrieve columns from database table ReviewTable, I changed the SQL query.
Rather than call if (rs.next()), you can use a while loop. That way you can iterate over all the rows in the Resultset.
Since you don't know how many rows there are in the ResultSet, use a List to store details from all the rows.
Since your class Product stores an array and not a List, I convert the Lists to arrays. Note that the code for converting List<Double> to double[] came from this SO question: How to cast from List to double[] in Java?
Now I have the three arguments I need in order to call the constructor of class Product. So I call the constructor and return the object that the constructor returned.
I have a rest service that take xml with 400_000 records, each record contain the following fields: code,type,price.
In DB (MySql )I have table named PriceData with 2_000_000 rows. The purpose of this rest is: select all PriceDatas from DB according to code,type from XML, replace price of each PriceData with price from XML, if there is no PriceData with this code,type create new with provided price.
Now it work as : select one PriceData from DB accroding to first record from XML, set new price or create new PriceData, save PriceData and these steps repeats 400_000 times.(It takes about 5 minutes)
I want to speed up this process.
First try:
Select 1000 elements step by step from PriceData, and when all elements will be selected update them:
Code:
private void updateAll(final List<XmlData> prices/*data from xml*/) {
int end= 1000;
int begin= 0;
final List<PriceData> models = new ArrayList<>();
while(end != prices.size() || begin !=end){
models.addAll(dao.findByPrices(prices.subList(begin,end)));
begin = end;
end +=1000;
}
final Map<String,XmlData> xmlData= prices.stream()
.collect(Collectors.toMap(this::keyForPriceDate,e->e));
final Map<String,PriceData> modelMap = models.stream()
.collect(Collectors.toMap(this::keyForRowModel,e->e));
final List<PriceData> modelsToSave = new ArrayList<>();
for(final String key : xmlData.keySet()){
final XmlData price = xmlData.get(key);
PriceData model = modelMap.get(key);
if(model == null){
model = onEmptyPriceData(price);
}
model.setPrice(price.getPrice());
modelsToSave.add(model);
}
modelService.saveAll(modelsToSave);
}
I convert two lists to maps to know does PriceData exist (keys for xmlData and modelMap created as (code+type))
findByPrices method create query in following format
select * from PriceData where (code =123 and type ='qwe') or (...)//and this `Or` repeats 1000 times
Now it takes 2 minutes.
Second try:
Select all PriceData from db (2 millions)
and use the algorithm above
It takes 3 minutes. First try is better but in future my rest can take 500_000 and I want to know which try will be better in this scenario or maybe there is the better way to do this task.
My select method
public List<PriceData> findBy(final List<XmlData> selectData) {
final StringBuilder query = new StringBuilder("SELECT * from PriceData ");
query.append("WHERE \n");
final Iterator<PriceRowSelectData> selectDataIterator = selectData.iterator();
while(selectDataIterator.hasNext()){
final PriceRowSelectData data = selectDataIterator.next();
query.append("( \n")
.append("productCode = "+ data.getProductId()+" \n")
.append(" AND type = "+ data.getPriceind()+" \n")
.append(" ) \n");
if(selectDataIterator.hasNext()){
query.append("OR \n");
}
}
final SearchResult<PriceRowModel> searchRes = search(query.toString());
/*
Here i use custom mapper that map list of result to my object
*/
return searchRes.getResult();
}
You should use the MySQL INSERT ... ON DUPLICATE KEY UPDATE statement, combined with JDBC batch processing. This of course assumes that code,type is the primary key, or at least a unique index.
private void updateAll(final List<XmlData> prices) throws SQLException {
String sql = "INSERT INTO PriceData (code, type, price)" +
" VALUES (?,?,?)" +
" ON DUPLICATE KEY" +
" UPDATE price = ?";
try (PreparedStatement stmt = this.conn.prepareStatement(sql)) {
int batchSize = 0;
for (XmlData price : prices) {
if (batchSize == 1000) { // flush batch every 1000
stmt.executeBatch();
batchSize = 0;
}
stmt.setInt (1, price.getCode());
stmt.setString (2, price.getType());
stmt.setBigDecimal(3, price.getPrice());
stmt.setBigDecimal(4, price.getPrice());
stmt.addBatch();
batchSize++;
}
if (batchSize != 0)
stmt.executeBatch();
}
}
You can twiddle the batch size, but not flushing will use a lot of memory. I think 1000 statements per batch is good, but I have no numbers backing that.
I am trying to update the field of an entry in an SQLiteDatabase using the db.update(...) method, but it seems the value is not stored. I've tried the convenience db.query(...) method right after the update method has been executed and found that the entry is still stored as before the update.
Is there some sort of background work that I must wait for before the query, or where am I going wrong? I am using a singleton extended SQLiteOpenHelper (dbHelper) as recommended in SQLite DB accessed from multiple threads and I've even tried getting a new readable instance of the db from the helper for the query in a new thread, as in the code below:
ContentValues deviceListEntry = new ContentValues();
deviceListEntry.put(DeviceListDBEntry.NODE_ID, nodeID);
...
...
String WHERE = DeviceListDBEntry.NODE_ID + " = ?";
final String[] WHERE_ARG = {String.valueOf(nodeID)};
SQLiteDatabase db = dbHelper.getWritableDatabase();
int listings = 0;
try {
//Update the device in the database DeviceList table
listings = db.update(
DeviceListDBEntry.TABLE_NAME,
deviceListEntry,
WHERE,
WHERE_ARG
);
} catch (Exception e) {
throw new ApiHandlerException("db.update(DeviceList, node " + nodeID + ")", e);
}
Log.e("updateDBdevice", " node " + device.getNodeID() + " listening = " + device.isListening());
final String[] TABLE_COLUMNS = {
DeviceListDBEntry.DEVICE_TYPE,
DeviceListDBEntry.INTERVIEWED,
DeviceListDBEntry.DEVICE_JSON
};
final String where = WHERE;
new Thread(new Runnable() {
#Override
public void run() {
SQLiteDatabase db2 = dbHelper.getReadableDatabase();
Cursor deviceEntry = db2.query(
DeviceListDBEntry.TABLE_NAME, //FROM DeviceList Table
TABLE_COLUMNS, //SELECT * columns
where, //WHERE nodeID =
WHERE_ARG, //args nodeID
null,
null,
null
);
if (!deviceEntry.moveToFirst()) throw new ApiHandlerException("DeviceListDB no entry found - WHERE nodeID = " + nodeID);
if (deviceEntry.getCount() > 1) throw new ApiHandlerException("DeviceListDB duplicate entries - WHERE nodeID = " + nodeID);
String deviceJson = deviceEntry.getString(deviceEntry.getColumnIndexOrThrow(DeviceListDBEntry.DEVICE_JSON));
Log.e("updateDBdevice retreive", " node " + nodeID + " JSON : " + deviceJson);
}
}).start();
I am using a Gson object to parse my device class to a JSON object which is stored in the DB. I know that this works when using the db.insert(...) method.
The query here is only there to see if the update was successful, because I found that explicit queries using other delayed threads (synchronised using a object lock and the same SQLiteOpenHelper) returned values that were not updated.
Is there an obvious thing I am missing or should I consider going to raw SQL commands on the db?
My mistake, I found that I had actually not added the updated JSON object to the new entry. Subsequently the deviceJson column of the listing did not update, but a db.update() was executed...
If "WHERE" clause has "text" column comparison then use single quotes around value. In your case try below line (notice single quotes around ?)
String WHERE = DeviceListDBEntry.NODE_ID + " = '?'";
I have a table in H2 DB
Order
--------
id (key)
MarketId1
MarketId2
MarketId3
ListName1
ListName2
ListName3
From XML I'm getting list of ListOrder
public final class ListOrder
{
public long listId;
public String Name;
}
So I have 3 prepared statements
"UPDATE Order set " + ListName1 + " = ? WHERE " + MarketId1 + " = ?"
"UPDATE Order set " + ListName2 + " = ? WHERE " + MarketId2 + " = ?"
"UPDATE Order set " + ListName3 + " = ? WHERE " + MarketId3 + " = ?"
The in a method I prepare a list of PreparedStament to execute
final PreparedStatement statement1 = connection.prepareStatement(QUERY1);
final PreparedStatement statement2 = connection.prepareStatement(QUERY2);
final PreparedStatement statement3 = connection.prepareStatement(QUERY3);
for (ListOrder listOrder: listOrders)
{
statement1.setString(1, listOrder.Name);
statement1.setLong(2, listOrder.listId);
statement1.addBatch();
statement2.setString(1, listOrder.Name);
statement2.setLong(2, listName.listId);
statement2.addBatch();
statement3.setString(1, listName.Name);
statement3.setLong(2, listOrder.listId);
statement3.addBatch();
}
return new ArrayList<PreparedStatement>(){{add(statement1); add(statement2); add(statement3);}};
I'm a SQL noob. Is there any better way of doing it? I assume that MarketId 1 2 3 could be the same. ListNames could be null (there will be at least one)
UPDATE:
In code I would write something like this (prob change to HashMap)
for (ListOrder listOrder: listOrders)
{
for(Order order : orders)
{
if(order.marketID1 == listOrder.listID)
order.listName1 = listOrder.Name; //break if no dups
if(order.marketID2 == listOrder.listID)
order.listName2 = listOrder.Name;
if(order.marketID3 == listOrder.listID)
order.listName3 = listOrder.Name;
}
}
You can use update comma separated
UPDATE <TABLE>
SET COL1 = <VAL1>,
COL2= <VAL2>
WHERE <CONDITION>
Is it this what you expect as one update query?
Unless you are trying to update the same record, then there is no way to do this easily or efficiently in a single query. Otherwise, assuming this is the desired result, you could use an OR (or an AND if that is desired) statement such as:
UPDATE Order
SET ListName1=?, ListName2=?, ListName3=?
WHERE MarketId1=? OR MarketId2=? OR MarketId3=?
You might also consider updating your table to use a one:many relationship which might make your queries easier. For example:
Order
--------
id (key)
name
etc
Market_List
--------
id (key)
order_id (fk)
market
listname