I have trouble with logical definition of the HashMap.
For example I create the following class to store some mandatory data, I just wanna know that is it good implementation or not? I used static HashMap because I need these HashMaps all over the time since my application is alive.
public abstract class DataTable {
private static HashMap<String, String[]> mainData = new HashMap<String, String[]>();
public static void putData(String[] data) {
// put some data
}
public static String[] getData(String alias) {
// return entered data with the given alias
}
}
Any suggestion would be appreciated...
Your static understanding is ok.
And your methods (setters and getters) should be
public static void putData(String key ,String[] data) {
mainData.put(key, data);
}
public static HashMap<String, String[]> getData(String alias) {
return mainData;
}
Of course ,obviously proper gauridng and exception handling is mandatory.
Update:As you mentioned in comment,that you are asking about thread safety on map use ConcurrentHashMap.
Map<String, String[]> mainData = new ConcurrentHashMap<String, String[]>();
Which is a
A hash table supporting full concurrency of retrievals and adjustable expected concurrency for updates.
A nice article on diff
Related
i have the following use case.
Xml files are written to a kafka topic which i want to consume and process via flink.
The xml attributes have to be renamed to match the database table columns. These renames have to be flexible and maintainable from outside the flink job.
At the end the attributes have to be written to the database.
Each xml document repesent a database record.
As a second step all some attributes of all xml documents from the last x minutes have to be aggregated.
As i know so far flink is capable of all the mentioned steps but i am lacking of an idea how to implement it corretly.
Currently i have implemented the kafka source, retrieve the xml document and parse it via custom MapFunction. There i create a POJO and store each attribute name and value in a HashMap.
public class Data{
private Map<String,String> attributes = HashMap<>();
}
HashMap containing:
Key: path.to.attribute.one Value: Value of attribute one
Now i would like to use the Broadcasting State to change the original attribute names to the database column names.
At this stage i stuck as i have my POJO data with the attributes inside the HashMap but i don't know how to connect it with the mapping via Broadcasting.
Another way would be to flatMap the xml document attributes in single records. This leaves me with two problems:
How to assure that attributes from one document don't get mixed with them from another document within the stream
How to merge all the attributes of one document back to insert them as one record into the database
For the second stage i am aware of the Window function even if i don't have understood it in every detail but i guess it would fit my requirement. The question on this stage would be if i can use more than one sink in one job while one would be a stream of the raw data and one of the aggregated.
Can someone help with a hint?
Cheers
UPDATE
Here is what i got so far - i simplified the code the XmlData POJO is representing my parsed xml document.
public class StreamingJob {
static Logger LOG = LoggerFactory.getLogger(StreamingJob.class);
public static void main(String[] args) throws Exception {
// set up the streaming execution environment
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
XmlData xmlData1 = new XmlData();
xmlData1.addAttribute("path.to.attribute.eventName","Start");
xmlData1.addAttribute("second.path.to.attribute.eventTimestamp","2020-11-18T18:00:00.000");
xmlData1.addAttribute("third.path.to.attribute.eventSource","Source1");
xmlData1.addAttribute("path.to.attribute.additionalAttribute","Lorem");
XmlData xmlData2 = new XmlData();
xmlData2.addAttribute("path.to.attribute.eventName","Start");
xmlData2.addAttribute("second.path.to.attribute.eventTimestamp","2020-11-18T18:00:01.000");
xmlData2.addAttribute("third.path.to.attribute.eventSource","Source2");
xmlData2.addAttribute("path.to.attribute.additionalAttribute","First");
XmlData xmlData3 = new XmlData();
xmlData3.addAttribute("path.to.attribute.eventName","Start");
xmlData3.addAttribute("second.path.to.attribute.eventTimestamp","2020-11-18T18:00:01.000");
xmlData3.addAttribute("third.path.to.attribute.eventSource","Source1");
xmlData3.addAttribute("path.to.attribute.additionalAttribute","Day");
Mapping mapping1 = new Mapping();
mapping1.addMapping("path.to.attribute.eventName","EVENT_NAME");
mapping1.addMapping("second.path.to.attribute.eventTimestamp","EVENT_TIMESTAMP");
DataStream<Mapping> mappingDataStream = env.fromElements(mapping1);
MapStateDescriptor<String, Mapping> mappingStateDescriptor = new MapStateDescriptor<String, Mapping>(
"MappingBroadcastState",
BasicTypeInfo.STRING_TYPE_INFO,
TypeInformation.of(new TypeHint<Mapping>() {}));
BroadcastStream<Mapping> mappingBroadcastStream = mappingDataStream.broadcast(mappingStateDescriptor);
DataStream<XmlData> dataDataStream = env.fromElements(xmlData1, xmlData2, xmlData3);
//Convert the xml with all attributes to a stream of attribute names and values
DataStream<Tuple2<String, String>> recordDataStream = dataDataStream
.flatMap(new CustomFlatMapFunction());
//Map the attributes with the mapping information
DataStream<Tuple2<String,String>> outputDataStream = recordDataStream
.connect(mappingBroadcastStream)
.process();
env.execute("Process xml data and write it to database");
}
static class XmlData{
private Map<String,String> attributes = new HashMap<>();
public XmlData(){
}
public String toString(){
return this.attributes.toString();
}
public Map<String,String> getColumns(){
return this.attributes;
}
public void addAttribute(String key, String value){
this.attributes.put(key,value);
}
public String getAttributeValue(String attributeName){
return attributes.get(attributeName);
}
}
static class Mapping{
//First string is the attribute path and name
//Second string is the database column name
Map<String,String> mappingTuple = new HashMap<>();
public Mapping(){}
public void addMapping(String attributeNameWithPath, String databaseColumnName){
this.mappingTuple.put(attributeNameWithPath,databaseColumnName);
}
public Map<String, String> getMappingTuple() {
return mappingTuple;
}
public void setMappingTuple(Map<String, String> mappingTuple) {
this.mappingTuple = mappingTuple;
}
}
static class CustomFlatMapFunction implements FlatMapFunction<XmlData, Tuple2<String,String>> {
#Override
public void flatMap(XmlData xmlData, Collector<Tuple2< String,String>> collector) throws Exception {
for(Map.Entry<String,String> entrySet : xmlData.getColumns().entrySet()){
collector.collect(new Tuple2<>(entrySet.getKey(), entrySet.getValue()));
}
}
}
static class CustomBroadcastingFunction extends BroadcastProcessFunction {
#Override
public void processElement(Object o, ReadOnlyContext readOnlyContext, Collector collector) throws Exception {
}
#Override
public void processBroadcastElement(Object o, Context context, Collector collector) throws Exception {
}
}
}
Here's some example code of how to do this using a BroadcastStream. There's a subtle issue where the attribute remapping data might show up after one of the records. Normally you'd use a timer with state to hold onto any records that are missing remapping data, but in your case it's unclear whether a missing remapping is a "need to wait longer" or "no mapping exists". In any case, this should get you started...
private static MapStateDescriptor<String, String> REMAPPING_STATE = new MapStateDescriptor<>("remappings", String.class, String.class);
#Test
public void testUnkeyedStreamWithBroadcastStream() throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(2);
List<Tuple2<String, String>> attributeRemapping = new ArrayList<>();
attributeRemapping.add(new Tuple2<>("one", "1"));
attributeRemapping.add(new Tuple2<>("two", "2"));
attributeRemapping.add(new Tuple2<>("three", "3"));
attributeRemapping.add(new Tuple2<>("four", "4"));
attributeRemapping.add(new Tuple2<>("five", "5"));
attributeRemapping.add(new Tuple2<>("six", "6"));
BroadcastStream<Tuple2<String, String>> attributes = env.fromCollection(attributeRemapping)
.broadcast(REMAPPING_STATE);
List<Map<String, Integer>> xmlData = new ArrayList<>();
xmlData.add(makePOJO("one", 10));
xmlData.add(makePOJO("two", 20));
xmlData.add(makePOJO("three", 30));
xmlData.add(makePOJO("four", 40));
xmlData.add(makePOJO("five", 50));
DataStream<Map<String, Integer>> records = env.fromCollection(xmlData);
records.connect(attributes)
.process(new MyRemappingFunction())
.print();
env.execute();
}
private Map<String, Integer> makePOJO(String key, int value) {
Map<String, Integer> result = new HashMap<>();
result.put(key, value);
return result;
}
#SuppressWarnings("serial")
private static class MyRemappingFunction extends BroadcastProcessFunction<Map<String, Integer>, Tuple2<String, String>, Map<String, Integer>> {
#Override
public void processBroadcastElement(Tuple2<String, String> in, Context ctx, Collector<Map<String, Integer>> out) throws Exception {
ctx.getBroadcastState(REMAPPING_STATE).put(in.f0, in.f1);
}
#Override
public void processElement(Map<String, Integer> in, ReadOnlyContext ctx, Collector<Map<String, Integer>> out) throws Exception {
final ReadOnlyBroadcastState<String, String> state = ctx.getBroadcastState(REMAPPING_STATE);
Map<String, Integer> result = new HashMap<>();
for (String key : in.keySet()) {
if (state.contains(key)) {
result.put(state.get(key), in.get(key));
} else {
result.put(key, in.get(key));
}
}
out.collect(result);
}
}
I've looked at so many examples but can't quite grasp this.
I need to create a method that inserts new values into already populated lists within my hashmap. I can't for the life of me figure out how to do. Can anyone help as well as explain how it works?
I've already created methods that populate the maps etc. I just can't figure out how to create a method that inserts just values for particular keys.
import java.util.*;
public class Singles
{
// instance variables - replace the example below with your own
private Map<String, List<String>> interests;
/**
* Constructor for objects of class Singles
*/
public Singles()
{
// initialise instance variables
super();
this.interests = new HashMap<>();
}
}
This is a multi-map.
public class MultiMap {
private Map<String, List<String>> multiMap = new HashMap<>();
public void put(String key, String value) {
List<String> values = (this.multiMap.containsKey(key) ? this.multiMap.get(key) : new ArrayList<>());
values.add(value);
this.multiMap.put(key, values);
}
}
What is the best way to avoid multiple parallel if-else loop. I tried with switch statement as well, but again that doesn't look readable. I have hundreds of such statements:
public static Map getKqvSecureNodeResponse(Sample secureNodeData, Map<String, Object> map) {
if(map.containsKey(Constants.NAME_KQV)) {
map.put(Constants.NAME_KQV, secureNodeData.getNodename());
}
if(map.containsKey(Constants.SPOV)) {
map.put(Constants.SPOV, secureNodeData.getOverride());
}
if(map.containsKey(Constants.SPEP)) {
map.put(Constants.SPEP, secureNodeData.getEnabledProtocol());
}
if(map.containsKey(Constants.SPTO)) {
map.put(Constants.SPTO, secureNodeData.getAuthTimeout());
}
if(map.containsKey(Constants.TLCN)) {
map.put(Constants.TLCN, secureNodeData.getCommonName());
}
if(map.containsKey(Constants.SEDT)) {
map.put(Constants.SEDT, secureNodeData.getEncryptData());
}
if(map.containsKey(Constants.TLCF)) {
map.put(Constants.TLCF, secureNodeData.getKeyCertLabel());
}
if(map.containsKey(Constants.TLCL)) {
map.put(Constants.TLCL, secureNodeData.getCipherSuites());
}
return map;
}
Please note that I have to invoke different getter of secureNodeData for every check.
For each Constants value (e.g. Constants.NAME_KQV), you can provide a Function<Sample, Object> (e.g. sample -> sample.getNodename()).
If you organised it in a structure like Map or enum (here, I used a enum), you could end up with a simple loop:
public static Map<String, Object> getKqvSecureNodeResponse(Sample secureNodeData, Map<String, Object> map) {
for (Constant constant : Constant.values()) {
final String name = constant.getName();
if (map.containsKey(name)) {
map.put(name, constant.getFunction().apply(secureNodeData));
}
}
return map;
}
The enum was defined as:
enum Constant {
NAME_KQV(Constants.NAME_KQV, Sample::getNodename);
// other definitions
final String name;
final Function<Sample, Object> function;
Constant(String name, Function<Sample, Object> function) {
this.name = name;
this.function = function;
}
public String getName() {
return name;
}
public Function<Sample, Object> getFunction() {
return function;
}
}
It seems like this method does a lot. (1) It's unclear why it overrides existing values. (2) The method name is obscure. (3) You are using a raw Map, replace it with Map<String, Object> at least, and figure out how to substitute the Object part. (4)
I feel rethinking the design would help much more than the above approach and these small corrections.
You can try to take advantage of method references:
public static Map getKqvSecureNodeResponse(Sample node, Map<String, Object> map) {
applyParam(Constants.NAME_KQV, map, node::getNodename);
applyParam(Constants.SPOV, map, node::getOverride);
// ...
}
public static void applyParam(String key, Map<String, Object> data, Supplier<Object> getter) {
if (data.containsKey(key)) {
data.put(key, getter.get());
}
}
Alternatively you can use Function references that are instance independent:
private static final Map<String, Function<Sample, Object>> MAPPING;
static {
MAPPING = new LinkedHashMap<>();
MAPPING.put(Constants.NAME_KQV, Sample::getNodename);
MAPPING.put(Constants.SPOV, Sample::getOverride);
}
public static Map getKqvSecureNodeResponse(Sample node, Map<String, Object> map) {
for (String key : MAPPING.keySet()) {
if (map.containsKey(key)) {
map.put(key, MAPPING.get(key).apply(node));
}
}
}
There are many ways how you can approach your specific use case, but method references in general makes developer's life much much easier.
Can ByteArrayOutputStream be stored into some other container like say HashMap?
If not how do I merge all my streams and then zip archive by entries into 1 file.
public class CFr {
private static HashMap<String, Object> fileEntries;
public static void setFileEntries(String fileNameEntry, Object fileEntry) {
CFr.fileEntries.put(fileNameEntry, fileEntry);
}
}
public void addDocx(CDb cd) {
CFr.setFileEntries((String)entryName, (ByteArrayOutputStream)bos);
}
And I get NullPointer on that setFileEntries line. Doesn't seem right, I was just assuming it is possible.
Yes, it can be stored, but you will need to initialize your HashMap for that purpose. In your case this
private static HashMap<String, Object> fileEntries;
is never initialized. You will need to do something like this:
private static HashMap<String, Object> fileEntries = new HashMap<String, Object>();
instead. This will fix your current issue.
I have a nested map:
Map<Integer, Map<Integer, Double>> areaPrices = new HashMap<Integer, Map<Integer, Double>>();
and this map is populated using the code:
while(oResult.next())
{
Integer areaCode = new Integer(oResult.getString("AREA_CODE"));
Map<Integer, Double> zonePrices = areaPrices.get(areaCode);
if(zonePrices==null)
{
zonePrices = new HashMap<Integer, Double>();
areaPrices.put(areaCode, zonePrices);
}
Integer zoneCode = new Integer(oResult.getString("ZONE_CODE"));
Double value = new Double(oResult.getString("ZONE_VALUE"));
zonePrices.put(zoneCode, value);
myBean.setZoneValues(areaPrices);
}
I want to use the value of this Map in another method of the same class. For that I have a bean.
How do I populate it on the bean, so that I can get the ZONE_VALUE in this other method
In my bean I added one new field as:
private Map<Integer, Map<Integer, Double>> zoneValues;
with getter and setter as:
public Map<Integer, Map<Integer, Double>> getZoneValues() {
return zoneValues;
}
public void setZoneValues(Map<Integer, Map<Integer, Double>> areaPrices) {
this.zoneValues = areaPrices;
}
What I am looking for to do in the other method is something like this:
Double value = myBean.get(areaCode).get(zoneCode);
How do I make it happen :(
I would like to suggest a different, hopefully more readable solution:
public class PriceMap {
private Map<Integer, Map<Integer, Double>> priceMap =
new HashMap<Integer, Map<Integer, Double>>();
// You'd use this method in your init
public Double setPrice(Integer areaCode, Integer zoneCode, Double price) {
if (!priceMap.containsKey(zoneCode)) {
priceMap.put(zoneCode, new HashMap<Integer, Double>());
}
Map<Integer, Double> areaMap = priceMap.get(zoneCode);
areaMap.put(areaCode, price);
}
public void getPrice(Integer areaCode, Integer zoneCode) {
if (!priceMap.containsKey(zoneCode)) {
// Eek! Exception or return null?
}
Map<Integer, Double> areaMap = priceMap.get(zoneCode);
return areaMap.get(areaCode);
}
}
I think this is a better, more readable abstraction which, very importantly, makes it easier for you or someone else to read after a few months.
EDIT Added get get
If you're stuck with a get(areaCode).get(zoneCode) (order reversed), but myBean is entirely yours, you could do something like:
public class MyBean {
// I suppose you have this already
private final Map<Integer, Map<Integer, Double>> priceMap =
new HashMap<Integer, Map<Integer, Double>>();
private class LooksLikeAMap implements Map<Integer, Double> {
private Integer areaCode = areaCode;
public LooksLikeAMap(Integer areaCode) {
this.areaCode = areaCode;
}
public Double get(Object zoneCode) {
if (!priceMap.containsKey(zoneCode)) {
// Eek! Exception or return null?
}
Map<Integer, Double> areaMap = priceMap.get(zoneCode);
return areaMap.get(areaCode);
}
// Implement other methods similarly
}
public Map<Integer, Double> get(Integer areaCode) {
return new LooksLikeAMap(areaCode);
}
}
OK, programming in a HTML textarea is not my strong suit, but the idea is clear.
Make some Map like structure backed by the complete data set, and initialize that
Map structure with the required AreaCode.
If the idea is not clear, post a comment fast as it's late here:)
EDIT
I am an idiot. I thought the data was zone first, then area while the get should be area first, then zone. In this case the Map already has the right structure, first area then zone, so this is not necessary. The get-get is by default if you make
public MyBean {
public Map<Integer, Double> get(Integer areaCode) {
return data.get(areaCode);
}
}
To start with, all you need is
myBean.getZoneValues(areaCode).get(zoneCode);
the while loop has an annoyance, you need to call myBean.setZoneValues(areaPrices);
out side the while loop
You can't directly control the second get() call because you have a nested Map, you'll need to return the appropriate nested Map to be able to do what you want. A getter like this should do it:
public Map<Integer, Double> get(Integer areaCode) {
return zoneValues.get(areaCode);
}
So when the client code calls get(areaCode) a map will be returned that they can then call get(zoneCode) on.
I'd suggest that you refactor to eliminate the nested Maps though, because you can't stop client code from changing the returned Map, the code is tough to read and you'll have problems if you want to add any more functionality - imagine that you want to provide a String description of an area code in future.
Something like a Map<Integer, AreaCode> where AreaCode is an object that contains what you currently have as a nested Map might be a good place to start.