We Keep Coding

Hazelcast MapLoader taking time to load the data from Hbase

We are using Hazelcast as our cached database in our project and HBase as the persistent database solution. We load the data from Hbase to Hazelcast using MapLoader when the application starts. Below is the snippet of the loadAll() function.
#Override
public synchronized Map<String, Object> loadAll(Collection<String> rows) {
final Map<String, Object> mapObject = new HashMap<String, Object>();
try {
if (!rows.isEmpty()) {
//Get Scanner from Hbase
final ResultScanner scanner = loadAllData(rows, this.mapName);
final byte[] family = this.mapName.getBytes();
final byte[] qualifier = this.mapName.getBytes();
//Load data from Hbase to Hazelcast
for (Result result = scanner.next(); result != null; result = scanner
.next()) {
mapObject
.put(Bytes.toString(result.getRow()),
convertToObject(result.getValue(family,
qualifier)));
}
LOGGER.logDebug(Constants.EXIT, methodName,
"All the data's are successfully loaded for Table "
+ this.mapName);
}
} catch (Exception e) {
throw new RuntimeException(e);
}
return mapObject;
}
We have a table with 350k entries (total data size of 300 mb) and its taking close to 1.5 hour to load the data. Is this normal behavior? Does Hazelcast usually take this much time to load the data?

Misbehavior with sharedpreferences

Before you judge me, I'd like to say I read theese:
Saving a List of Strings in Android with SharedPreferences
Misbehavior when trying to store a string set using SharedPreferences
getString(String key, String defValue) - Reference
But I still can't understand, can't get things work. I get totally misbehavior of my preferences. My code:
public static SharedPreferences sharedAppPreferences;
public static final String AppsListKey = "AppListKey";
public static final String AppsPreferences = "AppsPreferences";
public static ArrayList<String> packageNames;
public void chooseApps(View view) {
sharedAppPreferences = getSharedPreferences(AppsPreferences, Context.MODE_PRIVATE);
if (sharedAppPreferences.contains(AppsListKey)) {
Set<String> buffer = new LinkedHashSet<String>(sharedAppPreferences.getStringSet(AppsListKey, new LinkedHashSet<String>()));
packageNames = new ArrayList<String>(buffer);
} else {
packageNames = new ArrayList<String>();
}
PackageManager packageManager = getPackageManager();
int flags = PackageManager.GET_META_DATA | PackageManager.GET_SHARED_LIBRARY_FILES | PackageManager.GET_UNINSTALLED_PACKAGES;
List<ApplicationInfo> packageList = packageManager.getInstalledApplications(flags);
for (ApplicationInfo pack : packageList) {
if (((pack.flags & ApplicationInfo.FLAG_SYSTEM) == 1) || packageNames.contains(pack.loadLabel(packageManager).toString())) {
// System application or already in array
} else {
// Installed by user and isnt in array
packageNames.add(pack.loadLabel(packageManager).toString());
}
}
Editor editor = sharedAppPreferences.edit();
Set<String> buffer1 = new LinkedHashSet<String> (packageNames);
editor.putStringSet(AppsListKey, buffer1);
editor.commit();
//packageNames.clear();
//buffer1.clear();
buffer1 = new LinkedHashSet<String>(sharedAppPreferences.getStringSet(AppsListKey, new LinkedHashSet<String>()));
packageNames = new ArrayList<String>(buffer1);
AppList appList = new AppList();
appList.show(getSupportFragmentManager(), "AppList");
}
Why first time I run my app I get list like
[Skype, Facebook, Whatsapp, Twitter, Google+]
It's ok as long as app is running... But if I kill my app and restart I get now totally different list like
[Whatsapp, Google+, Skype, Twitter, Facebook]
Could someone explain me please what is here wrong?

The only difference in your list before and after, is the ordering...
To expand on what I have been talking about, I just realised your mistake..
LinkedHashSet is ordered. However, you are storing in
Set<String> buffer
Set<String> is not ordered...
So it jumbles it up again.
You need to store it in parameter of LinkedHashSet like below
LinkedHashSet<String> buffer = new LinkedHashSet<String>(sharedAppPreferences.getStringSet(AppsListKey, new LinkedHashSet<String>()));
(There are two occasions i see this needs to be changed. )
Edit:
One final thing you can do to help, is rather than create a new list, cast the existing one from shared prefs ...
LinkedHashSet<String> buffer = (LinkedHashSet<String>)sharedAppPreferences.getStringSet(AppsListKey, new LinkedHashSet<String>());
There are alternatives to HashSets here

Want to store a HashMap? [duplicate]

This question already has answers here:
How to store HashMap on Android?
(3 answers)
Closed 9 years ago.
I have a HashMap that has been generated in an activity and I want to store it using something similar to SharedPreferences. I've tried to find information and have been referenced to something called GSON but I'm not exactly sure what that is. If I have something like:
HashMap<String, String> hashMap = new HashMap<String, String> ();
hashMap.put("String 1", "Value 1");
hashMap.put("String 2", "Value 2");
How can I store hashMap so that I can then read it in another activity and use that information? It also needs to be stored should the user close the app.

Json is very similar to HasMap. To save a Json you have to use a key and value like HasMap:
JSONObject userDetail = new JSONObject();
userDetail.put("email", email);
userDetail.put("token", token);
After that you can use a FileWriter or FileInputStream to save it in file .json and you can get it from other activitys using a JSONParser.
For more information about json look this

Go with Victor's answer.
But if your hashMap is complex, like(hash inside hash of hash)
you can store it directly in a file and read it later:
Write to file:
public void saveHashToFile(HashMap<String, Object> hash) throws IOException{
String filePath = getApplicationContext().getFilesDir().getPath().toString() + "/your.properties";
File file = new File(filePath);
FileOutputStream f = new FileOutputStream(file);
ObjectOutputStream s = new ObjectOutputStream(f);
s.writeObject(getProperty_Global());
s.close();
}
Reading back from file:
public HashMap<String, Object> getLocalHashFromFile() throws OptionalDataException, ClassNotFoundException, IOException{
String filePath = getApplicationContext().getFilesDir().getPath().toString() + "/your.properties";
File file = new File(filePath);
FileInputStream f = new FileInputStream(file);
ObjectInputStream s = new ObjectInputStream(f);
HashMap<String, Object> hashFromFile=(HashMap<String, Object>) s.readObject();
s.close();
Log.e("hashfromfileLOCAL", ""+hashFromFile);
return hashFromFile;
}

Gson is a Java library for converting Java objects into JSON strings and vice-verse.
I had the same issue in my project, and used Gson for converting the HashMap into a String, saving it into SharedPreferences, and getting it back in another activity.
To store the Map:
SharedPreferences preferences = getSharedPreferences("com.your.package", MODE_PRIVATE);
Type genericType = new TypeToken<HashMap<String, String>>() {}.getType();
String serializedHashMap = Helpers.serializeWithJSON(your_hashmap, genericType);
preferences.edit().putString("Somename", serializedHashMap).commit();
serializeWithJSON():
public static String serializeWithJSON(Object o, Type genericType) {
Gson gson = new Gson();
return gson.toJson(o, genericType);
}
To deserialize:
Gson gson = new Gson();
Type genericType = new TypeToken<HashMap<String, String>>() {}.getType();
HashMap<String, String> hashMap = gson.fromJson(preferences.getString("Somename", "Errormessage"), genericType);
To persistently store it in a file, use amalBit's answer.

read/write files in Java

How can I read a file easily in Java if I have following file format:
a|dip
a|dop
c|nap
a|dip
b|dop
b|sip
a|tang
c|dig
c|nap
I want to get all words that belongs to "a", "b", and "c". What data structure I can use to read and store this information?
You can also suggest some good file formats (two column) that is easy to read/write in Java.
I know some of you may be thinking that what is the real problem that I want to solve, I have some complex employee related data. Current (poor) system generate some files and I am trying to process them to add them in database. The current files' format is bit complex (private), I cannot copy past here.

If you can use Google Guava (http://code.google.com/p/guava-libraries/) then you'll get a few handy classes (you can use some or all of these):
com.google.common.io.Files
com.google.common.io.LineProcessor<T>
com.google.common.base.Charsets
com.google.common.collect.Multimap<K,V>
com.google.common.collect.ArrayListMultimap<K,V>
For example you could write:
LineProcessor<Multimap<String, String>> processor =
new LineProcessor<Multimap<String, String>>() {
Multimap<String, String> processed = ArrayListMultimap.create();
public boolean processLine(String line) {
String parts[] = line.split("\\|", 2); // 2 keeps any | in the rest of the line
processed.put(parts[0], parts[1]);
return true; // keep going
}
public Multimap<String, String> getResult() {
return processed;
}
};
Multimap<String, String> result = Files.readLines(
new File("filename.txt"), Charsets.UTF_8, processor);

You can use Scanner to read the text file one line at a time and then you can use String.split("\\|") to separate the parts on that line. For storing the information, a Map<String,List<String>> might work.

I'd use this data structure:
Map<String, List<String>> map = new HashMap<String, List<String>>();
And parse the file like this:
File file = new File("words.txt");
Scanner scanner = new Scanner(file);
while (scanner.hasNext()) {
String next = scanner.next();
String[] parts = next.split("\\|");
String group = parts[0];
String word = parts[1];
List<String> list = map.get(group);
if (list == null) {
list = new ArrayList<String>();
map.put(group, list);
}
list.add(word);
}
So you could get the list of words for "a" like so:
for (String word : map.get("a")) {
System.out.println(word);
}

How to read and write a HashMap to a file?

I have the following HashMap:
HashMap<String,Object> fileObj = new HashMap<String,Object>();
ArrayList<String> cols = new ArrayList<String>();
cols.add("a");
cols.add("b");
cols.add("c");
fileObj.put("mylist",cols);
I write it to a file as follows:
File file = new File("temp");
FileOutputStream f = new FileOutputStream(file);
ObjectOutputStream s = new ObjectOutputStream(f);
s.writeObject(fileObj);
s.flush();
Now I want to read this file back to a HashMap where the Object is an ArrayList.
If i simply do:
File file = new File("temp");
FileInputStream f = new FileInputStream(file);
ObjectInputStream s = new ObjectInputStream(f);
fileObj = (HashMap<String,Object>)s.readObject();
s.close();
This does not give me the object in the format that I saved it in.
It returns a table with 15 null elements and the < mylist,[a,b,c] > pair at the 3rd element. I want it to return only one element with the values I had provided to it in the first place.
//How can I read the same object back into a HashMap ?
OK So based on Cem's note: This is what seems to be the correct explanation:
ObjectOutputStream serializes the objects (HashMap in this case) in whatever format that ObjectInputStream will understand to deserialize and does so generically for any Serializable object.
If you want it to serialize in the format that you desire you should write your own serializer/deserializer.
In my case: I simply iterate through each of those elements in the HashMap when I read the Object back from the file and get the data and do whatever I want with it. (it enters the loop only at the point where there is data).
Thanks,

You appear to be confusing the internal resprentation of a HashMap with how the HashMap behaves. The collections are the same. Here is a simple test to prove it to you.
public static void main(String... args)
throws IOException, ClassNotFoundException {
HashMap<String, Object> fileObj = new HashMap<String, Object>();
ArrayList<String> cols = new ArrayList<String>();
cols.add("a");
cols.add("b");
cols.add("c");
fileObj.put("mylist", cols);
{
File file = new File("temp");
FileOutputStream f = new FileOutputStream(file);
ObjectOutputStream s = new ObjectOutputStream(f);
s.writeObject(fileObj);
s.close();
}
File file = new File("temp");
FileInputStream f = new FileInputStream(file);
ObjectInputStream s = new ObjectInputStream(f);
HashMap<String, Object> fileObj2 = (HashMap<String, Object>) s.readObject();
s.close();
Assert.assertEquals(fileObj.hashCode(), fileObj2.hashCode());
Assert.assertEquals(fileObj.toString(), fileObj2.toString());
Assert.assertTrue(fileObj.equals(fileObj2));
}

I believe you´re making a common mistake. You forgot to close the stream after using it!
File file = new File("temp");
FileOutputStream f = new FileOutputStream(file);
ObjectOutputStream s = new ObjectOutputStream(f);
s.writeObject(fileObj);
s.close();

you can also use JSON file to read and write MAP object.
To write map object into JSON file
ObjectMapper mapper = new ObjectMapper();
Map<String, Object> map = new HashMap<String, Object>();
map.put("name", "Suson");
map.put("age", 26);
// write JSON to a file
mapper.writeValue(new File("c:\\myData.json"), map);
To read map object from JSON file
ObjectMapper mapper = new ObjectMapper();
// read JSON from a file
Map<String, Object> map = mapper.readValue(
new File("c:\\myData.json"),
new TypeReference<Map<String, Object>>() {
});
System.out.println(map.get("name"));
System.out.println(map.get("age"));
and import ObjectMapper from com.fasterxml.jackson and put code in try catch block

Your first line:
HashMap<String,Object> fileObj = new HashMap<String,Object>();
gave me pause, as the values are not guaranteed to be Serializable and thus may not be written out correctly. You should really define the object as a HashMap<String, Serializable> (or if you prefer, simpy Map<String, Serializable>).
I would also consider serializing the Map in a simple text format such as JSON since you are doing a simple String -> List<String> mapping.

I believe you're getting what you're saving. Have you inspected the map before you save it? In HashMap:
/**
* The default initial capacity - MUST be a power of two.
*/
static final int DEFAULT_INITIAL_CAPACITY = 16;
e.g. the default HashMap will start off with 16 nulls. You use one of the buckets, so you only have 15 nulls left when you save, which is what you get when you load.
Try inspecting fileObj.keySet(), .entrySet() or .values() to see what you expect.
HashMaps are designed to be fast while trading off memory. See Wikipedia's Hash table entry for more details.

Same data if you want to write to a text file
public void writeToFile(Map<String, List<String>> failureMessage){
if(file!=null){
try{
BufferedWriter writer=new BufferedWriter(new FileWriter(file, true));
for (Map.Entry<String, List<String>> map : failureMessage.entrySet()) {
writer.write(map.getKey()+"\n");
for(String message:map.getValue()){
writer.write(message+"\n");
}
writer.write("\n");
}
writer.close();
}catch (Exception e){
System.out.println("Unable to write to file: "+file.getPath());
e.printStackTrace();
}
}
}