What is the right way to persist a huge List<String> entity property with Spring jpa (postgresql db)? I've been converting it to byte[] which was stored as bytea in db , however it causes different formatting troubles (even simply rewriting the list).
List<String> can be as big as 10.000.000 entities. sometimes i just need to take the list, change a few elements and write it back.
As i understand, other options are:
use ElementCollection - which i think isn't the solution for such large lists.
Store it as large string with split characters - guess that isn't very efficient option either.
currently i use these methods to transform the list to/from byte[]:
public static List<String> bytesToStringListByInputStream(byte[] bytes) {
List<String> lines;
if (bytes == null) {
return lines;
}
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
DataInputStream in = new DataInputStream(bais);
try {
while (in.available() > 0) {
lines.add(in.readUTF());
}
} catch (IOException e) {
e.printStackTrace();
}
return lines;
}
public static byte[] arrayListToByteArray(List<String> list){
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputStream out = new DataOutputStream(baos);
for (String element : list) {
try {
out.writeUTF(element + System.lineSeparator());
} catch (IOException e) {
e.printStackTrace();
}
}
try {
out.flush();
} catch (IOException e) {
e.printStackTrace();
}
return baos.toByteArray();
}
entity:
#MappedSuperclass
#EntityListeners(AuditingEntityListener.class)
public abstract class UserFile implements Serializable {
...some fields...
#JsonProperty(access = JsonProperty.Access.WRITE_ONLY)
private byte[] fileData;
however in different cases after changing the list and persisting it, random characters appear in ouput lines - sometimes numbers or special symbols.
I guess i can find find a reason why it happenning, but in any case - is this a right way to persist a large lists (through byte[] jpa entity field and bytea postgresql field)?
Related
I have a file with some info how can I read all info?
Name names;
try (FileInputStream fileInputStream = new FileInputStream(file)) {
ObjectInputStream objectInputStream = new ObjectInputStream(fileInputStream);
names = (Name) objectInputStream.readObject();
} catch (IOException | ClassNotFoundException e) {
e.printStackTrace();
}
You have several solution, all depending on the input:
You can iterate until the stream is fully consumed: I think that is the worse solution out of those I provide you. It is worse because you are checking if EOF was reached, whilst you should know when you're done (eg: your file format is wrong).
Set<Name> result = new HashSet<>();
try {
for (;;) {
result.add((Name)objectInputStream.readObject());
}
} catch (EOFException e) {
// End of stream
}
return result;
When producing the input, serialize a collection and invoke readObject() on it. Serialization should be able to read the collection, as long as each object implements Serializable.
static void write(Path path, Set<Name> names) throws IOException {
try (OutputStream os = Files.newOutputStream(path);
ObjectOutputStream oos = new ObjectOutputStream(os)) {
oos.writeObject(names);
}
}
static Set<Name> read(Path path) throws IOException {
try (InputStream is = Files.newInputStream(path);
ObjectInputStream ois = new ObjectInputStream(is)) {
// WARN Files.newInputStream is not buffered; ObjectInputStream might
// be buffered (I don't remember).
return (Set<Name>) ois.readObject();
}
}
When producing the input, you can add a int indicating the number of object to read, and iterate over it: this is useful in case where you don't really care of the collection (HashSet). The resulting file will be smaller (because you won't have the HashSet metadata).
int result = objectInputStream.readInt();
Name[] names = new Name[result]; // do some check on result!
for (int i = 0; i < result; ++i) {
names[i] = (Name) objectInputStream.readObject();
}
Also, Set are good, but since they remove duplicate using hashCode()/equals() you may get less object if your definition of equals/hashCode changed after the fact (example: your Name was case sensitive and now it is not, eg: new Name("AA").equals(new Name("aa"))).
I have pretty large number of classes which are Serializable/Externalizable and unfortunately it happens quite often that I forget to serailize/desirialize a new field when adding it to a class or mispell readObject instead of readDouble. So I decided to write some sort of unit test which is ignored and won't run by maven-surefire-plugin. It's only for my own needs. It looks like this:
#Test(enabled = false)
public void testEquality(Object o){
String oPart = o.toString();
try (FileOutputStream fos = new FileOutputStream("/home/myusr/testser/" + oPart);
ObjectOutputStream ous = new ObjectOutputStream(fos)) {
ous.writeObject(o);
} catch (FileNotFoundException | IOException e) {
e.printStackTrace();
}
Object oo = null;
try (FileInputStream fis = new FileInputStream("/home/myusr/testser/" + oPart);
ObjectInputStream ois = new ObjectInputStream(fis)) {
Object oo = ois.readObject();
} catch (FileNotFoundException | IOException | ClassNotFoundException e) {
e.printStackTrace();
}
// Need to compare all non-transient fields of o and oo
}
But the question is how to compare all fields which are not transient with the same name. I would not want to override equals method just for testing purpose.
Is there a way to deal with it? Maybe there are some commons/guava libraries which can cope with that.
You can use reflection to do it starting with YourClaas.class.getDeclaredFields(). Then you have to filter the returned fields, retrieve the values from the objects and do the comparison.
Hi I need to store ArrayList in database of my project, for that I need some suggestion to store ArrayList into DB by means of some techniques like ArrayList to Byte[] and store byte[] in DB and retreive my byte[] from DB and convert into my original ArrayList in Java
I assume you're talking about Java's ArrayList (you didn't mention the language you have in mind).
In fact, there was a vary similar question: Serializing Arraylists
Just serialize a list, and insert it to BLOB column. Then you can fetch it from DB and deserialize it.
Edit:
This is a piece of code you can use to serialize your ArrayList:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
ObjectOutputStream ooStream = new ObjectOutputStream(baos);
ooStream.writeObject(myList);
ooStream.close();
} catch(IOException e) {
throw new RuntimeException(e);
} finally {
try {
baos.close();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
byte[] serializedData = baos.toByteArray();
Having the data as byte[] you can now store it in bytea / BLOB column.
I hope this helps.
I know there are a few methods for saving data in an android application, but I'm wondering what is the simplest and/or most effective, or in other words which method would win-out on a scale of complexity/rationality/performance.
Basically I just have two Class ArrayLists (ArrayLists of class objects, not primitive data types). One ArrayList's object's constructor takes three integers,the other four. I basically need to store the value of those integers (I have methods for each set up to return the integers either as strings or ints) with a way of telling what each one belonged to.
For instance, if I have:
arrayListOne.get(1).getNumbers() returning 1, 2, 3
arrayListTwo.get(1).getNumbers() returning 1, 2, 3, 4
and a whole heap of other indexes that would return different numbers, how can I store that data so when the app is closed and restarted it is reloaded and the values stay true to the indexes they were initialized at?
Writing it to internal storage is one solution. You can use the following as a static method inside a Util class:
Retrieve the ArrayList:
final static String OBJECT_1_LIST = "object_1_list";
static ArrayList<MyObject1> object1List = null;
static ArrayList<MyObject1> getObject1List(Context mContext) {
FileInputStream stream = null;
try {
stream = mContext.openFileInput(OBJECT_1_LIST);
ObjectInputStream din = new ObjectInputStream(stream);
object1List = (ArrayList<MyObject1>) din.readObject();
stream.getFD().sync();
stream.close();
din.close();
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e1) {
e1.printStackTrace();
}
if (object1List == null) {
object1List = new ArrayList<MyObject1>();
}
return object1List;
}
Similarly, to update the ArrayList:
private static void updateObject1List(Context mContext) {
FileOutputStream stream = null;
try {
stream = mContext.openFileOutput(OBJECT_1_LIST,
Context.MODE_PRIVATE);
ObjectOutputStream dout = new ObjectOutputStream(stream);
dout.writeObject(object1List);
stream.getFD().sync();
stream.close();
dout.flush();
} catch (IOException e) {
e.printStackTrace();
}
}
To add an item:
static void addToObject1list(Context mContext, MyObject1 obj) {
Utilities.getObject1List(mContext).add(obj);
Utilities.updateObject1List(mContext);
}
Add methods for removing an item and clearing the ArrayList.
You also need MyObject1 to implement Serializable:
public class MyObject1 implements Serializable {
....
....
}
I have found saving objects as JSON strings to be very simple and effective.
Check out the GSON library. It will convert from JSON to java and java back to JSON with very little fiddling.
https://code.google.com/p/google-gson/
I love GSON for doing this very thing.
I have serialized a HashTable<String,Object> object using an ObjectOutputStream. When serializing the object, I get no exception, but upon deserialization, the following exception occurs:
Exception in thread "main" java.io.InvalidClassException: java.lang.Long; local class
incompatible: stream classdesc serialVersionUID = 4290774032661291999, local class
serialVersionUID = 4290774380558885855
I no longer get the error when I remove all of the keys in the HashTable that have a value that is not a String (all of the key / value pairs I removed had a primitive type as their value).
What could be causing this error?
UPDATE - Here's the code
public static String serialize(Quiz quiz) throws IOException{
HashMap<String,Object> quizData = new HashMap<String,Object>();
quizData.put("version", 0); //int
quizData.put("name", quiz.getName()); //String
quizData.put("desc", quiz.getDesc()); //String
quizData.put("timelimitType", quiz.getTimelimitType()); //String
quizData.put("timelimit", quiz.getTimelimit()); //long
ArrayList<String> serializedQuestionsData = new ArrayList<String>();
for (Question question : quiz.getQuestions())
serializedQuestionsData.add(Question.serialize(question));
quizData.put("questions", serializedQuestionsData.toArray(new String[0])); //String[]
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos;
try { oos = new ObjectOutputStream(baos); } catch (IOException error){ throw error; }
try { oos.writeObject(quizData); } catch (IOException error){ throw error; }
return baos.toString();
}
#SuppressWarnings("unchecked")
public static Quiz deserialize(String serializedQuizData) throws IOException, ClassNotFoundException{
ByteArrayInputStream bais = new ByteArrayInputStream(serializedQuizData.getBytes());
ObjectInputStream ois;
try { ois = new ObjectInputStream(bais); } catch (IOException error){ throw error; }
HashMap<String,Object> quizData;
// Exception occurs on the following line!!
try { quizData = (HashMap<String,Object>) ois.readObject(); } catch (ClassNotFoundException error){ throw error; }
Quiz quiz;
if ((int) quizData.get("version") == 0){
quiz = new Quiz((String) quizData.get("name"),
(String) quizData.get("desc"),
(String) quizData.get("timelimitType"),
(long) quizData.get("timelimit"));
for (String serializedQuestionData : (String[]) quizData.get("questions"))
quiz.addQuestion(Question.deserialize(serializedQuestionData));
} else {
throw new UnsupportedOperationException("Unsupported version: \"" + quizData.get("version") + "\"");
}
return quiz;
}
The problem is that you're transforming a byte array output stream to a String using toString(). The toString() method simply uses the platform default encoding to transform the bytes (which do not represent characters at all but are purely binary data) into a String. This is thus a lossy operation, because your platform default encoding doesn't have a valid character for every possible byte.
You shouldn't use String to hold binary data. A String contains characters. If you really need a String, then encode the byte array using a Hexadecimal or Base64 encoder. Otherwise, simply use a byte array to hold your binary data:
public static byte[] serialize(Quiz quiz) throws IOException{
...
ByteArrayOutputStream baos = new ByteArrayOutputStream();
...
return baos.toByteArray();
}
#SuppressWarnings("unchecked")
public static Quiz deserialize(byte[] serializedQuizData) throws IOException, ClassNotFoundException{
ByteArrayInputStream bais = new ByteArrayInputStream(serializedQuizData);
...
return quiz;
}
The only explanation I can think of is that is that something is corrupting your object stream between you reading it and writing it. The serialVersionID in "the local class) (4290774380558885855) is standard across all Java implementations that try to be compatible with Java (tm). The source code for java.lang.Long says that that serial version id has not changed since Java 1.0.2.
If you need further help, you will need to provide an SSCCE that covers both creation and reading of the serialized object.