We Keep Coding

Persist data to the DB with message queue

We have a gRPC server that inserts the data into the CockRoachDB and the data is coming from a Spring Boot micro-service.
This is my code to persist in the CRDB database:
#Service
#Transactional(propagation = Propagation.REQUIRED, rollbackFor = Exception.class)
public class CockroachPersister {
private static final String X_AMZN_REQUESTID = "x-amzn-RequestId";
private static final String X_AMZN_RESPONSE = "x-amzn-Response";
private static final String PUTITEM = "PutItem";
private static final String GETITEM = "GetItem";
private static final String DELETEITEM = "DeleteItem";
private static final String UPDATEITEM = "UpdateItem";
public <T extends Message> T save(final String requestBody, final String action, final String tableName) {
T t = null;
try {
List<GRPCMapper> lGRPCMapper = ServiceMapper.getServices(action,tableName);
for (GRPCMapper grpcMapper : lGRPCMapper) {
System.out.println("grpcMapper.getClassName() ==> "+grpcMapper.getClassName());
Class<?> className = Class.forName(grpcMapper.getClassName());
Class<?> implementedClassType = Class.forName(grpcMapper.getImplementedClass());
Method userMethod = implementedClassType.getDeclaredMethod(grpcMapper.getServiceName(), className);
System.out.println("userMethod\t" + userMethod.getName());
t = (T) userMethod.invoke(null, ProtoUtil.getInstance(requestBody, grpcMapper.getProtoType()));
System.out.printf("Service => %s row(s) Inserted \n", t.getAllFields().toString());
}
} catch (Exception e) {
e.printStackTrace();
}
return t;
}
}
If the initial insertion failed, I would like to try at least 3 TIMES before we can log the error. How do I implement that?
A solution that use message queue will be also acceptable.

what is this following error about in spring rest webservice api?

I'am creating a restapi , i am using java spring and i'am getting the following error.
Error:
org.springframework.dao.EmptyResultDataAccessException: Incorrect result size: expected 1, actual 0
My daoImpl class
#Override
public String getLoginDetails(VendorLogin vendorlogin) {
String getVendorData = "select vendor_ID from vendor_login where vendor_ID= ?
and password=?";
String name =null;
try{
name = (String) jdbcTemplate.queryForObject(getVendorData,new Object[]{
vendorlogin.getVendorLoginId(), vendorlogin.getPassWord()}, String.class);
}catch(Exception e){
e.printStackTrace();
}
return name;
}
my controller
#RequestMapping(value = Constants.REQ_MAP_LOGIN,
method = RequestMethod.POST,
produces = MediaType.APPLICATION_JSON_VALUE)
public String vendorloginMethodPost(#RequestBody VendorLogin vendoridlogin) {
String message = Constants.EMPTY_STRING;
String id = dao.getLoginDetails(vendoridlogin);
String password = dao.getLoginDetails(vendoridlogin);
if (id == null && password==null) {
message = "login FAIL";
}else{
message =" login Successfully";
}
return message;
}

SOLUTION
#Override
public String getLoginDetails(VendorLogin vendorlogin) {
String getVendorData = "select vendor_ID from vendor_login where vendor_ID= ? and password=?";
try {
name = (String) jdbcTemplate.queryForObject(
getVendorData,
new Object[]{vendorlogin.getVendorLoginId(), vendorlogin.getPassWord()},
new RowMapper<YourVendorObject>() {
public UserAttempts mapRow(ResultSet rs, int rowNum) throws SQLException {
// we suppose that your vendor_ID is String in DB
String vendor_ID = rs.getString("vendor_ID");
// if you wanna return the whole object use setters and getters
// from rs.getInt ... rs.getString ...
return vendor_ID;
}
});
return name;
} catch (EmptyResultDataAccessException e) {
return null;
}
}
public class EmptyResultDataAccessException extends IncorrectResultSizeDataAccessException
Data access exception thrown when a result was expected to have at least one row (or element) but zero rows (or elements) were actually returned.
The problem is, Spring throws an EmptyResultDataAccessException, instead of returning a null when record not found :
JdbcTemplate .java
package org.springframework.jdbc.core;
public class JdbcTemplate extends JdbcAccessor implements JdbcOperations {
//...
public <T> T queryForObject(String sql, Object[] args,
RowMapper<T> rowMapper) throws DataAccessException {
List<T> results = query(sql, args, new RowMapperResultSetExtractor<T>(rowMapper, 1));
return DataAccessUtils.requiredSingleResult(results);
}
DataAccessUtils.java
package org.springframework.dao.support;
public abstract class DataAccessUtils {
//...
public static <T> T requiredSingleResult(Collection<T> results)
throws IncorrectResultSizeDataAccessException {
int size = (results != null ? results.size() : 0);
if (size == 0) {
throw new EmptyResultDataAccessException(1);
}
if (results.size() > 1) {
throw new IncorrectResultSizeDataAccessException(1, size);
}
return results.iterator().next();
}
check it here : source

try {
String getVendorData = "select vendor_ID from vendor_login where vendor_ID= ? and password=?";
String name =null;
name = (String) jdbcTemplate.queryForObject(getVendorData,new Object[]{vendorlogin.getVendorLoginId(), vendorlogin.getPassWord()}, String.class);
} catch (EmptyResultDataAccessException e) {
return null;
}

BeanUtils.setProperty method sets null a BigDecimal field

I'm having a problem using the BeanUtils.setProperty method.
I'm using this JAR:
<dependency>
<groupId>commons-beanutils</groupId>
<artifactId>commons-beanutils</artifactId>
<version>1.9.3</version>
</dependency>
I run a MySQL query that returns one record and I'm mapping the resultset to a JavaBean that I've made.
Here you have the main class.
public class QueryTester {
public static void viewTable(Connection con) throws SQLException, InstantiationException, IllegalAccessException, InvocationTargetException {
Statement stmt = null;
String query = "SELECT * FROM Books WHERE code = 'AA00'";
try {
stmt = (Statement) con.createStatement();
ResultSet rs = stmt.executeQuery(query);
ResultSetMapper<Books> rsMapper = new ResultSetMapper<Books>();
List<Books> list = rsMapper.mapResultSetToObject(rs, Books.class);
} catch (SQLException e) {
e.printStackTrace();
} finally {
if (stmt != null) {
stmt.close();
}
}
}
public static void main(String[] args) {
Connection conn = null;
String url = "jdbc:mysql://localhost/dbname";
String driver = "com.mysql.jdbc.Driver";
String userName = "root";
String password = "root";
try {
Class.forName(driver).newInstance();
conn = (Connection) DriverManager.getConnection(url,userName,password);
viewTable(conn);
conn.close();
} catch (Exception e) {
System.out.println("NO CONNECTION");
}
}
}
And this is the method that uses the BeanUtils.setProperty method.
public class ResultSetMapper<T> {
public List<T> mapResultSetToObject(ResultSet rs, Class<T> outputClass) throws InstantiationException, SQLException, IllegalAccessException, InvocationTargetException {
List<T> outputList = new ArrayList<T>();
if (rs == null) {
return outputList;
}
if (!outputClass.isAnnotationPresent(Entity.class)) {
throw new InstantiationException("Entity notation not present.");
}
ResultSetMetaData rsmd = rs.getMetaData();
// retrieve data fields from output class
Field[] fields = outputClass.getDeclaredFields();
while (rs.next()) {
T bean = (T) outputClass.newInstance();
for (int iterator = 0; iterator < rsmd.getColumnCount(); iterator++) {
String columnName = rsmd.getColumnName(iterator + 1);
Object columnValue = rs.getObject(iterator + 1);
for (Field field : fields) {
if (field.isAnnotationPresent(Column.class)) {
Column column = field.getAnnotation(Column.class);
if (column.name().equalsIgnoreCase(columnName) && columnValue != null) {
BeanUtils.setProperty(bean, field.getName(), columnValue);
break;
}
}
}
}
outputList.add(bean);
}
return outputList;
}
}
mapResultSetToObject method returns a List with one element that is correct but the bean is set in a wrong way.
The fields code and bookDescription are set right but kPrice field is set null instead of 3.000 that is the value from database.
I run this code in debug mode and "columnValue" variable's value is 3.000 but the setProperty method doesn't set the right value and the value remains null.
Here you have my Java Bean.
#Entity
public class Books {
#Column(name="code")
private String code;
#Column(name="book_description")
private String bookDescription;
#Column(name="kPrice")
private BigDecimal kPrice;
public Books() {}
public Books(String code, String bookDescription, BigDecimal kPrice){
this.code = code;
this.bookDescription = bookDescription;
this.kPrice = kPrice;
}
/* Getters and setters */
...
}
And this is the MySQL table and the record.
CREATE TABLE `Books` (
`code` varchar(4) NOT NULL,
`book_description` varchar(50) NOT NULL DEFAULT '',
`kPrice` decimal(10,4) NOT NULL DEFAULT '1.0000',
PRIMARY KEY (`code`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
INSERT INTO dbname.Books (code, book_description, kPrice) VALUES('AA00', 'Description example', 3.0000);
Why I get this behaviour? What am I missing?
Thanks in advance

Are you sure which the name of setters/getters is the same of property?
In some case, the problem is that.
See my example below:
#Entity
public class Books {
#Column(name="code")
private String code;
#Column(name="book_description")
private String bookDescription;
#Column(name="kPrice")
private BigDecimal kPrice;
public Books() {}
public Books(String code, String bookDescription, BigDecimal kPrice){
this.code = code;
this.bookDescription = bookDescription;
this.kPrice = kPrice;
}
public void setKPrice ( Bigdecimal kPrice) // and not setkPrice or setPrice..
{
this.kPrice = kPrice;
}
public BigDecimal getKPrice () // and not getkPrice or getPrice..
{
return this.kPrice;
}
}

My Storm Topology neither working(not generating output) nor failing (not generating errors or exceptions)

I have a topology in which I am trying to count word occurrences which are being generated by SimulatorSpout (not real Stream) and after that write to MySQL database table, the table scheme is very simple:
Field | Type | ...
ID | int(11) | Auto_icr
word | varchar(50) |
count | int(11) |
But I am facing weird problem(as I beforementioned)
I successfully submitted The Topology to my Storm Cluster which consists of 4 supervisors, and I can see the flow of the Topology in Storm Web UI
(no exceptions) but when I checked the MySQL table, to my surprise, the table is empty...
ANY COMMENTS, SUGGESTIONS ARE WELCOME...
Here are spouts and bolts:
public class MySQLConnection {
private static Connection conn = null;
private static String dbUrl = "jdbc:mysql://192.168.0.2:3306/test?";
private static String dbClass = "com.mysql.jdbc.Driver";
public static Connection getConnection() throws SQLException, ClassNotFoundException {
Class.forName(dbClass);
conn = DriverManager.getConnection(dbUrl, "root", "qwe123");
return conn;
}
}
============================= SentenceSpout ===============================
public class SentenceSpout extends BaseRichSpout{
private static final long serialVersionUID = 1L;
private boolean _completed = false;
private SpoutOutputCollector _collector;
private String [] sentences = {
"Obama delivered a powerfull speech against USA",
"I like cold beverages",
"RT http://www.turkeyairline.com Turkish Airlines has delayed some flights",
"don't have a cow man...",
"i don't think i like fleas"
};
private int index = 0;
public void open (Map config, TopologyContext context, SpoutOutputCollector collector) {
_collector = collector;
}
public void nextTuple () {
_collector.emit(new Values(sentences[index]));
index++;
if (index >= sentences.length) {
index = 0;
Utils.waitForSeconds(1);
}
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("sentence"));
}
public void ack(Object msgId) {
System.out.println("OK: " + msgId);
}
public void close() {}
public void fail(Object msgId) {
System.out.println("FAIL: " + msgId);
}
}
============================ SplitSentenceBolt ==============================
public class SplitSentenceBolt extends BaseRichBolt {
private static final long serialVersionUID = 1L;
private OutputCollector _collector;
public void prepare (Map config, TopologyContext context, OutputCollector collector) {
_collector = collector;
}
public void execute (Tuple tuple) {
String sentence = tuple.getStringByField("sentence");
String httpRegex = "((https?|ftp|telnet|gopher|file)):((//)|(\\\\))+[\\w\\d:##%/;$()~_?\\+-=\\\\\\.&]*";
sentence = sentence.replaceAll(httpRegex, "").replaceAll("RT", "").replaceAll("[.|,]", "");
String[] words = sentence.split(" ");
for (String word : words) {
if (!word.isEmpty())
_collector.emit(new Values(word.trim()));
}
_collector.ack(tuple);
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
}
=========================== WordCountBolt =================================
public class WordCountBolt extends BaseRichBolt {
private static final long serialVersionUID = 1L;
private HashMap<String , Integer> counts = null;
private OutputCollector _collector;
private ResultSet resSet = null;
private Statement stmt = null;
private Connection _conn = null;
private String path = "/home/hduser/logOfStormTops/logger.txt";
String rLine = null;
public void prepare (Map config, TopologyContext context, OutputCollector collector) {
counts = new HashMap<String, Integer>();
_collector = collector;
}
public void execute (Tuple tuple) {
int insertResult = 0;
int updateResult = 0;
String word = tuple.getStringByField("word");
//----------------------------------------------------
if (!counts.containsKey(word)) {
counts.put(word, 1);
try {
insertResult = wordInsertIfNoExist(word);
if (insertResult == 1) {
_collector.ack(tuple);
} else {
_collector.fail(tuple);
}
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
}
} else {
//-----------------------------------------------
counts.put(word, counts.get(word) + 1);
try {
// writing to db
updateResult = updateCountOfExistingWord(word);
if (updateResult == 1) {
_collector.ack(tuple);
} else {
_collector.fail(tuple);
}
// Writing to file
BufferedWriter buffer = new BufferedWriter(new FileWriter(path));
buffer.write("[ " + word + " : " + counts.get("word") + " ]");
buffer.newLine();
buffer.flush();
buffer.close();
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
System.out.println("{word-" + word + " : count-" + counts.get(word) + "}");
}
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
}
// *****************************************************
public int wordInsertIfNoExist(String word) throws ClassNotFoundException, SQLException {
String query = "SELECT word FROM wordcount WHERE word=\"" + word + "\"";
String insert = "INSERT INTO wordcount (word, count) VALUES (\"" + word + "\", 1)";
_conn = MySQLConnection.getConnection();
stmt = _conn.createStatement();
resSet = stmt.executeQuery(query);
int res = 0;
if (!resSet.next()) {
res = stmt.executeUpdate(insert);
} else {
System.out.println("Yangi qiymatni kirityotrganda nimadir sodir bo'ldi");
}
resSet.close();
stmt.close();
_conn.close();
return res;
}
public int updateCountOfExistingWord(String word) throws ClassNotFoundException, SQLException {
String update = "UPDATE wordcount SET count=count+1 WHERE word=\"" + word + "\"";
_conn = MySQLConnection.getConnection();
stmt = _conn.createStatement();
int result = stmt.executeUpdate(update);
//System.out.println(word + "'s count has been updated (incremented)");
resSet.close();
stmt.close();
_conn.close();
return result;
}
}
========================= WordCountTopology ==============================
public class WordCountTopology {
private static final String SENTENCE_SPOUT_ID = "sentence-spout";
private static final String SPLIT_BOLT_ID = "split-bolt";
private static final String COUNT_BOLT_ID = "count-bolt";
private static final String TOPOLOGY_NAME = "NewWordCountTopology";
#SuppressWarnings("static-access")
public static void main(String[] args) throws AlreadyAliveException, InvalidTopologyException {
SentenceSpout spout = new SentenceSpout();
SplitSentenceBolt splitBolt = new SplitSentenceBolt();
WordCountBolt countBolt = new WordCountBolt();
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout(SENTENCE_SPOUT_ID, spout, 2);
builder.setBolt(SPLIT_BOLT_ID, splitBolt, 4).shuffleGrouping(SENTENCE_SPOUT_ID);
builder.setBolt(COUNT_BOLT_ID, countBolt, 4).fieldsGrouping(SPLIT_BOLT_ID, new Fields("word"));
Config config = new Config();
config.setMaxSpoutPending(100);
config.setDebug(true);
StormSubmitter submitter = new StormSubmitter();
submitter.submitTopology(TOPOLOGY_NAME, config, builder.createTopology());
}
}

It is because the _collector.ack(tuple) is not being called when there is exception thrown. When there are too many pending tuple, spout will stop sending new tuples. Try throwing out RuntimeException instead of printStackTrace.

Working with ResultSets in Java

I've ran into a problem of having to run a number of different queries on the DB (different return types, different number of columns, etc).
While writing that i started to wonder if there's a proper way of writing a helper function.
It seemed that it's really easy to write a function that returns a ResultSet.
However since it a) doesn't close connection b) doesn't close the result set it seems as a possibly working, but improper solution. Is there any place to dump in all results so that they can be returned safely.
(Only thing i could come up with, is just returning a 2D string array (after converting all data to strings) and then converting it all back)
EDIT : Sorry for not writing clear, was wondering if there's any way to just store the result of the query as is (don't need to modify it) without writing a separate method for every possible return type.
The idea behind a 2d string list is being able to store the query values as is.
Col1 Row1 | Col2 Row1 | Col3 Row1
Col1 Row2 | Col2 Row2 | Col3 Row2
EDIT 2 Thank you for replies, i guess i'll just write a small parser for it.

You shouldn't be returning resultSets, you should read the results from the resultset into some kind of container object. A ResultSet is a wrapper around a database cursor, it goes away when the connection closes. It's something you read from and close right away, not something you can pass around your application.
Look at how spring-jdbc does it. You implement a resultSetMapper that is passed to the method on the JdbcTemplate.
Several observations:
You don't need to use Spring to use spring-jdbc. However, I see very little value in reimplementing this stuff yourself.
It's not the job of the code that reads the ResultSet to open and close connections, that needs to be elsewhere.

I'd recommend looking at Spring JDBC. Don't write such a thing yourself. It's already been done, and quite well.
For example, I don't like your idea of returning a List of Strings. You lose a lot of info that way. I'd return a Map of Lists (column view) or List of Maps (row view).
If you must, here are some database utilities that would get you started.
package persistence;
import java.sql.*;
import java.util.*;
/**
* util.DatabaseUtils
* User: Michael
* Date: Aug 17, 2010
* Time: 7:58:02 PM
*/
public class DatabaseUtils {
/*
private static final String DEFAULT_DRIVER = "oracle.jdbc.driver.OracleDriver";
private static final String DEFAULT_URL = "jdbc:oracle:thin:#host:1521:database";
private static final String DEFAULT_USERNAME = "username";
private static final String DEFAULT_PASSWORD = "password";
*/
/*
private static final String DEFAULT_DRIVER = "org.postgresql.Driver";
private static final String DEFAULT_URL = "jdbc:postgresql://localhost:5432/party";
private static final String DEFAULT_USERNAME = "pgsuper";
private static final String DEFAULT_PASSWORD = "pgsuper";
*/
private static final String DEFAULT_DRIVER = "com.mysql.jdbc.Driver";
private static final String DEFAULT_URL = "jdbc:mysql://localhost:3306/party";
private static final String DEFAULT_USERNAME = "party";
private static final String DEFAULT_PASSWORD = "party";
public static void main(String[] args) {
long begTime = System.currentTimeMillis();
String driver = ((args.length > 0) ? args[0] : DEFAULT_DRIVER);
String url = ((args.length > 1) ? args[1] : DEFAULT_URL);
String username = ((args.length > 2) ? args[2] : DEFAULT_USERNAME);
String password = ((args.length > 3) ? args[3] : DEFAULT_PASSWORD);
Connection connection = null;
try {
connection = createConnection(driver, url, username, password);
DatabaseMetaData meta = connection.getMetaData();
System.out.println(meta.getDatabaseProductName());
System.out.println(meta.getDatabaseProductVersion());
String sqlQuery = "SELECT PERSON_ID, FIRST_NAME, LAST_NAME FROM PERSON ORDER BY LAST_NAME";
System.out.println("before insert: " + query(connection, sqlQuery, Collections.EMPTY_LIST));
connection.setAutoCommit(false);
String sqlUpdate = "INSERT INTO PERSON(FIRST_NAME, LAST_NAME) VALUES(?,?)";
List parameters = Arrays.asList("Foo", "Bar");
int numRowsUpdated = update(connection, sqlUpdate, parameters);
connection.commit();
System.out.println("# rows inserted: " + numRowsUpdated);
System.out.println("after insert: " + query(connection, sqlQuery, Collections.EMPTY_LIST));
} catch (Exception e) {
rollback(connection);
e.printStackTrace();
} finally {
close(connection);
long endTime = System.currentTimeMillis();
System.out.println("wall time: " + (endTime - begTime) + " ms");
}
}
public static Connection createConnection(String driver, String url, String username, String password) throws ClassNotFoundException, SQLException {
Class.forName(driver);
if ((username == null) || (password == null) || (username.trim().length() == 0) || (password.trim().length() == 0)) {
return DriverManager.getConnection(url);
} else {
return DriverManager.getConnection(url, username, password);
}
}
public static void close(Connection connection) {
try {
if (connection != null) {
connection.close();
}
} catch (SQLException e) {
e.printStackTrace();
}
}
public static void close(Statement st) {
try {
if (st != null) {
st.close();
}
} catch (SQLException e) {
e.printStackTrace();
}
}
public static void close(ResultSet rs) {
try {
if (rs != null) {
rs.close();
}
} catch (SQLException e) {
e.printStackTrace();
}
}
public static void rollback(Connection connection) {
try {
if (connection != null) {
connection.rollback();
}
} catch (SQLException e) {
e.printStackTrace();
}
}
public static List<Map<String, Object>> map(ResultSet rs) throws SQLException {
List<Map<String, Object>> results = new ArrayList<Map<String, Object>>();
try {
if (rs != null) {
ResultSetMetaData meta = rs.getMetaData();
int numColumns = meta.getColumnCount();
while (rs.next()) {
Map<String, Object> row = new HashMap<String, Object>();
for (int i = 1; i <= numColumns; ++i) {
String name = meta.getColumnName(i);
Object value = rs.getObject(i);
row.put(name, value);
}
results.add(row);
}
}
} finally {
close(rs);
}
return results;
}
public static List<Map<String, Object>> query(Connection connection, String sql, List<Object> parameters) throws SQLException {
List<Map<String, Object>> results = null;
PreparedStatement ps = null;
ResultSet rs = null;
try {
ps = connection.prepareStatement(sql);
int i = 0;
for (Object parameter : parameters) {
ps.setObject(++i, parameter);
}
rs = ps.executeQuery();
results = map(rs);
} finally {
close(rs);
close(ps);
}
return results;
}
public static int update(Connection connection, String sql, List<Object> parameters) throws SQLException {
int numRowsUpdated = 0;
PreparedStatement ps = null;
try {
ps = connection.prepareStatement(sql);
int i = 0;
for (Object parameter : parameters) {
ps.setObject(++i, parameter);
}
numRowsUpdated = ps.executeUpdate();
} finally {
close(ps);
}
return numRowsUpdated;
}
}

You can write helper functions that parse a ResultSet and convert it into an ArrayList or an array or even the fields of an object. For instance, lets say you have a table of orders and then a query returns all of the rows of that table for a particular user (customer). We could then do something like this:
static List<Order> parseOrder(ResultSet rs) {
ArrayList<Order> orderList = new ArrayList<>();
while(rs.next() ) {
Order order = new Order();
order.setID(rs.getInt(1));
order.setCustomerID(rs.getInt(2));
order.setItemName(rs.getString(3));
orderList.add(order);
}
return orderList;
}
Simply turning the result set into an array of an array of Objects would be more general, but probably less useful.
I would leave it up to the calling function to close this ResultSet and possible the PreparedStatement (or Statement) and database connection.