I am trying to cache some data in storm bolt, but not sure if this is right way to do it or not. In below class employee id and employe name are cached to a hash map. For this a database call has been made to Employee table to select all employees and populate a hash map in prepare method (is this right place to initialize map?).
After some logging it turns out (while running storm topology), topology is making multiple database connections and initializing map multiple times. Ofcourse I want to avoid this, that is why I want to cache the result so that it does not go to database everytime. Please help?
public class TestBolt extends BaseRichBolt {
private static final long serialVersionUID = 2946379346389650348L;
private OutputCollector collector;
private Map<String, String> employeeIdToNameMap;
private static final Logger LOG = Logger.getLogger(TestBolt.class);
#Override
public void execute(Tuple tuple) {
String employeeId = tuple.getStringByField("employeeId");
String employeeName = employeeIdToNameMap.get(employeeId);
collector.emit(tuple, new Values(employeeId, employeeName));
collector.ack(tuple);
}
#Override
public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
// TODO Auto-generated method stub
this.collector = collector;
try {
employeeIdToNameMap = createEmployeIdToNameMap();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
#Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields(/*some fields*/));
}
private Map<String, String> createEmployeIdToNameMap() throws SQLException {
final Map<String, String> employeeIdToNameMap = new HashMap<>();
final DatabaseManager dbm = new PostgresManager();
final String query = "select id, name from employee;";
final Connection conn = dbm.createDefaultConnection();
final ResultSet result = dbm.executeSelectQuery(conn, query);
while(result.next()) {
String employeId = result.getString("id");
String name = result.getString("name");
employeeIdToNameMap.put(employeId, name);
}
conn.close();
return employeeIdToNameMap;
}
}
SOLUTION
I created synchronized map and its working fine for me
private static Map<String, String> employeeIdToNameMap = Collections
.synchronizedMap(new HashMap<String, String>());
Since you have multiple bolt tasks, you can mark employeeIdToNameMap static and volatile. Initialize the map in prepare like this -
try {
synchronized(TestBolt.class) {
if (null == employeeIdToNameMap) {
employeeIdToNameMap = createEmployeIdToNameMap();
}
}
} catch (SQLException e) {
...
}
Related
I have the following code using sqlite3 and java which runs without errors:
public class SqLiteDB {
public static void main(final String[] args) {
final SqLiteDB theDatabase = SqLiteDB.initSqLiteDB();
theDatabase.addParametersRow("test", "beam process");
theDatabase.addParametersRow("test", "beam process 1");
theDatabase.addParametersRow("test", "beam process 2");
theDatabase.addParametersRow("test1", "beam process 1");
theDatabase.addParametersRow("test1", "beam process 2");
theDatabase.addParametersRow("test2", "beam process");
theDatabase.addParametersRow("test2", "beam process 1");
theDatabase.addParametersRow("test3", "beam process");
theDatabase.addGroupStatements();
for (final Entry<String, String> result : theDatabase.getParametersAll().entrySet()) {
System.out.print(result.getKey() + ", " + result.getValue() + "\n");
}
}
private static final Logger LOGGER = LoggerFactory.getLogger(SqLiteDB.class);
private Connection dbConnection = null;
private static SqLiteDB myInstance = null;
// *****************************************************************************************************************
// Public methods
// *****************************************************************************************************************
public static SqLiteDB initSqLiteDB() {
if (myInstance == null) {
myInstance = new SqLiteDB();
}
return myInstance;
}
public void addParametersRow(final String parameterName, final String beamProcessName) {
final String sqlInsert = "INSERT INTO PARAMETERS (PARAMETER_NAME, BEAM_PROCESS) VALUES ('" + parameterName
+ "', '" + beamProcessName + "');";
try (Statement insertStatement = dbConnection.createStatement()) {
insertStatement.execute(sqlInsert);
} catch (final SQLException ex) {
ErrorLogger.logError(ex.getStackTrace(), ex);
}
}
public Map<String, String> getParametersAll() {
final String sqlGetParameterCount = "SELECT * FROM PARAMETERS";
final Map<String, String> queryResults = new TreeMap<>();
try (Statement queryParameterCount = dbConnection.createStatement()) {
final ResultSet resultSet = queryParameterCount.executeQuery(sqlGetParameterCount);
getClass();
while (resultSet.next()) {
queryResults.put(resultSet.getString(1), resultSet.getString(2));
}
resultSet.close();
} catch (final SQLException ex) {
ErrorLogger.logError(ex.getStackTrace(), ex);
}
return queryResults;
}
// *****************************************************************************************************************
// Private methods
// *****************************************************************************************************************
private SqLiteDB() {
try {
dbConnection = DriverManager.getConnection("jdbc:sqlite::memory:");
} catch (final SQLException ex) {
ErrorLogger.logError(ex.getStackTrace(), ex);
}
createTableAvailableParameters();
}
private void createTableAvailableParameters() {
if (dbConnection != null) {
final String sqlCreateTable = "CREATE TABLE IF NOT EXISTS PARAMETERS (\n"
+ " PARAMETER_NAME text NOT NULL, BEAM_PROCESS text NOT NULL, PRIMARY KEY (PARAMETER_NAME, BEAM_PROCESS));";
try (final Statement statement = dbConnection.createStatement()) {
statement.execute(sqlCreateTable);
} catch (final SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
private void addGroupStatements() {
final String sqlGetParameterCount = "SELECT PARAMETER_NAME, COUNT(PARAMETER_NAME) FROM PARAMETERS GROUP BY PARAMETER_NAME HAVING COUNT(PARAMETER_NAME) > 1";
final Map<String, Integer> queryResults = new HashMap<>();
try (Statement queryParameterCount = dbConnection.createStatement()) {
final ResultSet resultSet = queryParameterCount.executeQuery(sqlGetParameterCount);
getClass();
while (resultSet.next()) {
queryResults.put(resultSet.getString(1), resultSet.getInt(2));
}
resultSet.close();
} catch (final SQLException ex) {
ErrorLogger.logError(ex.getStackTrace(), ex);
}
for (final Entry<String, Integer> resultRow : queryResults.entrySet()) {
if (resultRow.getValue() > 1) {
addParametersRow(resultRow.getKey(), "*");
}
}
}
}
When I run the code it produces the following output:
test, *
test1, *
test2, *
test3, beam process
I do not understand where my rows with "test(1,2), beam process(1,2)" are? Why / where are they lost?
Stackoverflow does not accept post with a lot of code. I wouldn't know how to truncate the code without omitting details which might be necessary. That's why I have to put in some more useless text. I hope it helps to make the question postable.
The loss happens in the function getParametersAll(). In this function, you iterate over the available results and add them to a Map. However, if duplicate values occur, you override the already read results in the map. Thus, you only get the last created beam process for each of the parameters.
One way to solve this would be to create a Map<String, List<String>> and return this map instead.
Map<String, List<String>> queryResults = new TreeMap<>();
while (resultSet.next()) {
if (!queryResults.contains(resultSet.getString(1)) {
queryResults.put(resultSet.getString(1), new ArrayList<>());
}
queryResults.get(resultSet.getString(1)).add(resultSet.getString(2));
}
I hope, this answers your question, otherwise I have probably misunderstood your problem.
Based on Illedhar's answer - thanks to that - I implemented the method getParametersAll using the Guava Multimap interface. Here the working code:
public Multimap<String, String> getParametersAll() {
final String sqlGetParameterCount = "SELECT * FROM PARAMETERS";
final Multimap<String, String> queryResults = ArrayListMultimap.create();
try (Statement queryParameterCount = dbConnection.createStatement()) {
final ResultSet resultSet = queryParameterCount.executeQuery(sqlGetParameterCount);
getClass();
while (resultSet.next()) {
queryResults.put(resultSet.getString(1), resultSet.getString(2));
}
resultSet.close();
} catch (final SQLException ex) {
ErrorLogger.logError(ex.getStackTrace(), ex);
}
return queryResults;
}
The output in the main method needs to be changed sightly too:
for (final Entry<String, String> result : theDatabase.getParametersAll().entries()) {
System.out.print(result.getKey() + ", " + result.getValue() + "\n");
}
For cudos please vote 1+ for Illedhar.
I'm using this way to call the class and object still can't figure out what is wrong in getting the concept right
public class MMTUtil{
private static Map<String, String> domainDocumentationMap = null;
static{
domainDocumentationMap = new TreeMap<String, String>();
}
public static Map<String, String> getDomainDocumentationMap() {
return domainDocumentationMap;
}
public static void setDomainDocumentationMap(Map<String, String> domainDocumentationMap) {
MMTUtil.domainDocumentationMap = domainDocumentationMap;
}
How shall I use the setDomainDocumentation and getDomainDocumentation effectively with the code so that my Treemap has the following set of values
(objectType + objectname,DocumentationLink)
public UMRResultObject insertDocumentation(UMRDocumentationDTO documentationDTO)
{
Session session = UMRHibernateUtil.getUmrSession();
Transaction tx = null;
documentationLink = null;
objectName = null;
objectType = null;
try{
tx = session.beginTransaction();
dao.insertDocumentation(documentationDTO, session);
MMTUtil.getDomainDocumentationMap().put(objectName.getDomainName()+objectType.getDomainType(),documentationLink.getDocumentationLink());
tx.commit();
ro.setSuccess(true);
ro.getMessages().add("Record Inserted Successfully");
}
}
I retrieve values from a database, create a new Transaction Object and add it to an ArrayList<Transaction>, which I then return.
The problem is that everytime returnList.add(t); is called, instead of just adding the Transaction, it also replaces all old Transactions with the new one.
Where is the error that causes this behaviour?
public ArrayList<Transaction> getTransactions(long intervall, Map<String, String> transactionFields) {
connect();
ArrayList<Transaction> returnList = new ArrayList<Transaction>();
Statement sta;
ZonedDateTime now = ZonedDateTime.now(ZoneOffset.UTC);
now = now.minusSeconds(intervall);
try {
sta = conn.createStatement();
String Sql = "...";
ResultSet rs = sta.executeQuery(Sql);
while (rs.next()) {
Transaction t = new Transaction(rs.getString("No"), transactionFields);
t.set("AgentName", rs.getString("cname"));
returnList.add(t);
}
} catch (SQLException e) {
...
}
disconnect();
return returnList;
}
Here is the Transaction class:
public class Transaction {
private Map<String, String> fields;
public Transaction(String number, Map<String, String> transactionFields) {
fields = transactionFields;
fields.put("Number", number);
}
public void set(String field, String value) {
fields.put(field, value);
}
public String get(String field) {
return fields.get(field);
}
public Map<String, String> getFieldMap() {
return fields;
}
#Override
public String toString() {
return fields.toString();
You are using the same Map in all your Transaction instances.
Instead, pass in a new one each time:
Transaction t = new Transaction(rs.getString("No"), new HashMap<String, String>());
or just create the Map inside your Transaction class.
I want to fetch node and its information using label and index using cypher query but still i get null value in variable "result" in CypherQuery() method.
NewCypherQuery.java (bean)
public class NewCypherQuery {
private static final String DB_PATH = "/var/lib/neo4j/data/";
private static String resultString;
private static String columnsString, nodeResult, rows = "", query;
private static ExecutionResult result;
private static ExecutionEngine engine;
private static GraphDatabaseService db;
private static Node amad = null, pari = null, sona = null;
private static Relationship rel;
private static IndexDefinition inxamd, inxpri;
private static Label amd,pri;
public static void callAllMethods() {
clearDbPath();
setUp();
createNodes();
CypherQuery();
}
public static void CypherQuery() {
try (Transaction ignored = db.beginTx();) {
result = engine.execute("MATCH (m:inxamd)-->(n:inxpri) USING INDEX m:inxamd(name) USING INDEX n:inxpri(name) WHERE m.name = 'Amad' AND n.name= 'Pari' RETURN m");
for (Map<String, Object> row : result) {
resultString = engine.execute("MATCH (m:inxamd)-->(n:inxpri) USING INDEX m:inxamd(name) USING INDEX n:inxpri(name) WHERE m.name = 'Amad' AND n.name= 'Pari' RETURN m").dumpToString();
System.out.println(resultString);
}
} catch (Exception e) {
e.printStackTrace();
}
}
private static void setUp() {
try {
db = new GraphDatabaseFactory().newEmbeddedDatabase(DB_PATH);
try (Transaction tx = db.beginTx()) {
engine = new ExecutionEngine(db);
Schema schema = db.schema();
inxamd = schema.indexFor(amd).on("name").create();
inxpri = schema.indexFor(pri).on("name").create();
tx.success();
}
} catch (Exception e) {
e.printStackTrace();
}
}
private static void createNodes() {
try (Transaction tx = db.beginTx();) {
amad = db.createNode();
amad.setProperty("name", "Amad");
amad.setProperty("age", 24);
amad.setProperty("edu", "mscit");
pari = db.createNode();
pari.setProperty("name", "Pari");
pari.setProperty("age", 20);
pari.setProperty("edu", "mscit");
sona = db.createNode();
sona.setProperty("name", "Sona");
sona.setProperty("age", 21);
sona.setProperty("edu", "mscit");
rel = amad.createRelationshipTo(pari, RelTypes.KNOWS);
rel.setProperty("rel", "friend");
rel = pari.createRelationshipTo(sona, RelTypes.KNOWS);
rel.setProperty("rel", "friend");
System.out.println("Nodes created.....");
tx.success();
} catch (Exception e) {
e.printStackTrace();
}
}
private static void clearDbPath() {
try {
deleteRecursively(new File(DB_PATH));
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
InsertNodes.java (Servlet):-
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
// TODO Auto-generated method stub
NewCypherQuery ncq=new NewCypherQuery();
ncq.callAllMethods();
}
index.jsp
<form method="post" action="InsertNodes">
<input type="text" name="txtname" value="Hello World !!!!!"></input>
<input type="submit" value="Neo4j World"></input>
</form>
You haven't really defined values for the two labels amd (inxamd) and pri (inxpri) and nor have you assigned them to any nodes created.
You can either implement the Label class and assign a name to a label such as "inxamd" or use DynamicLabel.
Then, assign the label to your node using
amad.addLabel(thelabel);
Unrelated to your issue above, a label name is usually a descriptive string indicating what set the node belongs to e.g. Person, Dog
And in most cases you don't need to explicitly provide the index hint (USING INDEX)
I have a LinkedHashMap which fills with data from db with loop "for" string by string and when I try to show the first or the last String, the method can show me only the last String in log. But in application listViewContent is filled fully. So I don't understand why I can't see any string that I want. I need to collect all strings I get from db and compare them in future.
How can I collect all strings and what method should I call to show the string I want to see?Unfortunately I can only retrieve one (and the last instead of the first) string.
Here is my example code :
protected void onCreate(Bundle saveInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.main);
FirstMethod();
}
public FirstMethod() {
SecondMethod newMethod = .. // getting data from the second method
}
public SecondMethod() {
public void onResponseReceived(String result) {
try {
...
if (posts != null) {
for (WallPostItem post : posts) { // this loop
//create new map for a post
Map<String, Object> map = new LinkedHashMap<String, Object>();
map.put(ATTRIBUTE_NAME_TEXT, post.text);
PictureItem postPicture = new PictureItem();
map.put(ATTRIBUTE_NAME_IMAGE, postPicture);
map.put(ATTRIBUTE_NAME_DATE, post.date);
sAdapter.notifyDataSetChanged();
};
};
...
List<Map.Entry<String, Object>> list = new ArrayList<Map.Entry<String, Object>>(GlobalMap.entrySet());
Map.Entry<String, Object> firstInsertedEntry = list.get(0);
Log.w("FirstEntryOfMap",""+firstInsertedEntry); // this log shows me the last string instead of the first
}
if (isRefresh) {
isRefresh = false;
lvSimple.setSelectionAfterHeaderView();
}
} catch (Exception e) {
Log.d("exceptions", "problem in get wall post task after post execute: " + e.toString());
}
}
You aren't putting your values into a List, you are putting them into a Map (that preserves key order). I would suggest you create a POJO class,
class MyAttribute {
final String postName;
final PictureItem postPicture;
final Date postDate;
public MyAttribute(String postName, PictureItem postPicture, Date postDate) {
this.postName = postName;
this.postPicture = postPicture;
this.postDate = postDate;
}
public String getPostName() {
return postName;
}
public Date getPostDate() {
return postDate;
}
public PictureItem getPostPicture() {
return postPicture;
}
}
Then you could create a
List<MyAttribute> myAttributes = new ArrayList<>();