I have offline JSON definitions (in assets folder) - and with them I create my data model. It has like 8 classes which all inherit (extend) one abstract Model class.
Would it be better solution if I parse the JSON and keep the model in memory (more or less everything is Integer or String) through the whole life cycle of the App or would it be smarter if I parse the JSON files as they are needed?
thanks
Parsing the files and storing all the data in the memory will definitely give you a speed advantage. The problem with this solution is that if your application will go to back-ground (the user receives a phone call or just leaves the app by his will), no one can guarantee that the data will stay intact in memory.
This data can be clear by the GC if the system decided that it needs more memory.
This means that when the user comes back to the application and if you relay on the fact that the data is in the memory you might face an exception. So you need to consider this situation.
And from that point of you it is good to store you data on a file that can be parsed at a desired time, even thought this might be a slower solution.
Another solution you may look at is to parse this data at first application start-up to an SQLite DB and use it from there, or even store it in the DB in the first place. This will give you the advantages of both worlds, you would not have to parse the data before using it, and you will have a quick access to it using a Cursor and you are not facing the problem of data deletion in case of insufficient memory in the system.
I'd read all the file content at once and keep it as a static String somewhere in my application that is available to all application components (SingleTone Pattern) since usually Maintaining a small string in the memory is much cheaper than opening and closing files frequently.
To solve the GC point #Emil pointed out you can write your code something like this:
public class DataManager {
private static String myData;
public static String getData(Context context){
if(myData == null){
loadData(context);
}
return myData;
}
private static void LoadData(Context context){
context.getAssets().
try {
BufferedReader reader = new BufferedReader(
new InputStreamReader(getAssets().open("data.txt"), "UTF-8"));
StringBuilder builder = new StringBuilder();
do {
String mLine = reader.readLine();
builder.append(mLine);
} while (mLine != null)
reader.close();
myData = builder.toString();
} catch (IOException e) {
}
}
}
And from any class in your application that has a valid Context reference:
String data = DataManager.getData(context);
Related
I was learning to make quiz app from online and it is going well. I was wondering instead of reading json from assets , it will be wise to read from online such that question can be added or changes accordingly and user don't have to update app.
Here is the JSON Structure.
{"questions" : [{"category":"general","question": "Grand Central Terminal, Park Avenue, New York is the world's", "choices": ["largest railway station","highest railway station","longest railway station","None of the above"], "correctAnswer":0},
{"category":"science","question": "Entomology is the science that studies", "choices": ["Behavior of human beings","Insects","The origin and history of technical and scientific terms","the formation of rocks"], "correctAnswer":1},
{"category":"science", "question":"What is known as the 'master gland' of the human body?", "choices":["Thyroid gland","Pituitary gland","Pineal gland","Pancreas"],"correctAnswer":1}
]}
and the code to read from assets is
private String loadJSONFromAsset() {
String json = null;
try {
InputStream is = mContext.getAssets().open("questionsJSON.json");
int size = is.available();
byte[] buffer = new byte[size];
is.read(buffer);
is.close();
json = new String(buffer, "UTF-8");
} catch (IOException ex) {
ex.printStackTrace();
return null;
}
return json;
}
I would like to show progress loading dialog when next question loads and any help will be largely appreciated. Thanks in advance.
Best option will be using REST APIs, Get data from Server/Database, which can be edited anytime from anywhere
You can learn to use Node js, it is not hard and it is based on JavaScript.
For getting JSON from APIs you can use Retrofit
Learning and implementing these things will be a bit hard if you are beginner but it will be the best option for long run
hope this helped!
Maybe consider using two different threads (or Runnables), one thread for downloading the JSON content and the other thread for displaying the GUI. For example, take a look at this: Stackoverflow Post
The solution involved making a Runnable that would first start downloading the data from the online website and then update the current progress onto the GUI thread as it is downloading. He uses the BufferedInputStream class so he can use a while loop to read the data in, update the number of bytes downloaded, get the current progress, and then display the results. I suppose you can do something similar here by using a while loop, and then checking if the download is finished. If so, you can close the display.
In Java I am using a large String for <Html> codes which meant to create a complete designed email body.
eg.
String msg=
"<html>"+
<BODY CONTENT>
"</html>";
Problem is I am getting error "constant string too long”.
I need some ideas if anyone of you have solved this issue or faced this.
There is limitation the Constant string but one can have a much larger String in memory as long as it is created at run-time such as by reading it through a resource.
You can try this solution.
private static String msg = null;
public static String getMyString() throws IOException {
if (null == msg) {
try (BufferedReader br = new BufferedReader(new InputStreamReader(MyClass.class.getResourceAsStream("msg.txt")))) {
msg = br.lines().collect(Collectors.joining("\n"));
}
}
return msg;
}
You can call it and save it in another string :
String str = getMyString();
System.out.println("str = " + str);
or you can build your string with the string builder.
StringBuilder sb = new StringBuilder(100);
sb.append("<html>")
.append("<body>").append(bodycontent)
.append("</body>")
.append("</html>");
String result = sb.toString();
Hope this is helpful. cheers
There is not so many things we can do when facing constant string too long error.
Constants are constrained to 64K elements per single String entry, but you can split your exceeding constant in a couple of smaller than 64K ones as a workaround.
In terms of software design, at the other hand, the idea of working with complete email bodies as just Strings is not ultimately perfect. Usually developers are using template engines for such purposes and do externalize email bodies to a separate files rather than String constants. Please see Chunk template engine as example which fits well into Android app, but there are lots of another such as Velocity or Freemarker. Template engine lets you clearly separate static email content from dynamically-populated pieces, separate your application data model from it's html representation, and maintain valid architecture of your software
Being not aware of exact reason which prevents you from making a file instead of constant, there are lots of best practices to solve typical file-connected issues starting from embedding files into your jar as resources and ending in encrypting them to avoid unwanted leakage. Just ask another question on SO here.
Try to use StringBuilder, e.g.:
StringBuilder msg = new StringBuilder(here_put_size); //change string size, default value is 16 characters
msg.append("<html>");
msg.append("<BODY CONTENT>");
msg.append("</html>");
return msg.toString();
Is there a way to use StAX and JAX-B to create an index and then get quick access to an XML file?
I have a large XML file and I need to find information in it. This is used in a desktop application and so it should work on systems with few RAM.
So my idea is this: Create an index and then quickly access data from the large file.
I can't just split the file because it's an official federal database that I want to use unaltered.
Using a XMLStreamReader I can quickly find some element and then use JAXB for unmarshalling the element.
final XMLStreamReader r = xf.createXMLStreamReader(filename, new FileInputStream(filename));
final JAXBContext ucontext = JAXBContext.newInstance(Foo.class);
final Unmarshaller unmarshaller = ucontext.createUnmarshaller();
r.nextTag();
while (r.hasNext()) {
final int eventType = r.next();
if (eventType == XMLStreamConstants.START_ELEMENT && r.getLocalName().equals("foo")
&& Long.parseLong(r.getAttributeValue(null, "bla")) == bla
) {
// JAX-B works just fine:
final JAXBElement<Foo> foo = unmarshaller.unmarshal(r,Foo.class);
System.out.println(foo.getValue().getName());
// But how do I get the offset?
// cache.put(r.getAttributeValue(null, "id"), r.getCursor()); // ???
break;
}
}
But I can't get the offset. I'd like to use this to prepare an index:
(id of element) -> (offset in file)
Then I should be able use the offset to just unmarshall from there: Open file stream, skip that many bytes, unmarshall.
I can't find a library that does this. And I can't do it on my own without knowing the position of the file cursor. The javadoc clearly states that there is a cursor, but I can't find a way of accessing it.
Edit:
I'm just trying to offer a solution that will work on old hardware so people can actually use it. Not everyone can afford a new and powerful computer. Using StAX I can get the data in about 2 seconds, which is a bit long. But it doesn't require RAM. It requires 300 MB of RAM to just use JAX-B. Using some embedded db system would just be a lot of overhead for such a simple task. I'll use JAX-B anyway. Anything else would be useless for me since the wsimport-generated classes are already perfect. I just don't want to load 300 MB of objects when I only need a few.
I can't find a DB that just needs an XSD to create an in-memory DB, which doesn't use that much RAM. It's all made for servers or it's required to define a schema and map the XML. So I assume it just doesn't exist.
You could work with a generated XML parser using ANTLR4.
The Following works very well on a ~17GB Wikipedia dump /20170501/dewiki-20170501-pages-articles-multistream.xml.bz2 but I had to increase heap size using -xX6GB.
1. Get XML Grammar
cd /tmp
git clone https://github.com/antlr/grammars-v4
2. Generate Parser
cd /tmp/grammars-v4/xml/
mvn clean install
3. Copy Generated Java files to your Project
cp -r target/generated-sources/antlr4 /path/to/your/project/gen
4. Hook in with a Listener to collect character offsets
package stack43366566;
import java.util.ArrayList;
import java.util.List;
import org.antlr.v4.runtime.ANTLRFileStream;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.tree.ParseTreeWalker;
import stack43366566.gen.XMLLexer;
import stack43366566.gen.XMLParser;
import stack43366566.gen.XMLParser.DocumentContext;
import stack43366566.gen.XMLParserBaseListener;
public class FindXmlOffset {
List<Integer> offsets = null;
String searchForElement = null;
public class MyXMLListener extends XMLParserBaseListener {
public void enterElement(XMLParser.ElementContext ctx) {
String name = ctx.Name().get(0).getText();
if (searchForElement.equals(name)) {
offsets.add(ctx.start.getStartIndex());
}
}
}
public List<Integer> createOffsets(String file, String elementName) {
searchForElement = elementName;
offsets = new ArrayList<>();
try {
XMLLexer lexer = new XMLLexer(new ANTLRFileStream(file));
CommonTokenStream tokens = new CommonTokenStream(lexer);
XMLParser parser = new XMLParser(tokens);
DocumentContext ctx = parser.document();
ParseTreeWalker walker = new ParseTreeWalker();
MyXMLListener listener = new MyXMLListener();
walker.walk(listener, ctx);
return offsets;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
public static void main(String[] arg) {
System.out.println("Search for offsets.");
List<Integer> offsets = new FindXmlOffset().createOffsets("/tmp/dewiki-20170501-pages-articles-multistream.xml",
"page");
System.out.println("Offsets: " + offsets);
}
}
5. Result
Prints:
Offsets: [2441, 10854, 30257, 51419 ....
6. Read from Offset Position
To test the code I've written class that reads in each wikipedia page to a java object
#JacksonXmlRootElement
class Page {
public Page(){};
public String title;
}
using basically this code
private Page readPage(Integer offset, String filename) {
try (Reader in = new FileReader(filename)) {
in.skip(offset);
ObjectMapper mapper = new XmlMapper();
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
Page object = mapper.readValue(in, Page.class);
return object;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
Find complete example on github.
I just had to solve this problem, and spent way too much time figuring it out. Hopefully the next poor soul who comes looking for ideas can benefit from my suffering.
The first problem to contend with is that most XMLStreamReader implementations provide inaccurate results when you ask them for their current offsets. Woodstox however seems to be rock-solid in this regard.
The second problem is the actual type of offset you use. You have to use char offsets if you need to work with a multi-byte charset, which means the random-access retrieval from the file using the provided offsets is not going to be very efficient - you can't just set a pointer into the file at your offset and start reading, you have to read through until you get to the offset (that's what skip does under the covers in a Reader), then start extracting. If you're dealing with very large files, that means retrieval of content near the end of the file is too slow.
I ended up writing a FilterReader that keeps a buffer of byte offset to char offset mappings as the file is read. When we need to get the byte offset, we first ask Woodstox for the char offset, then get the custom reader to tell us the actual byte offset for the char offset. We can get the byte offset from the beginning and end of the element, giving us what we need to go in and surgically extract the element from the file by opening it as a RandomAccessFile, which means it's super fast at any point in the file.
I created a library for this, it's on GitHub and Maven Central. If you just want to get the important bits, the party trick is in the ByteTrackingReader.
Some people have commented about how this whole thing is a bad idea and why would you want to do it? XML is a transport mechanism, you should just import it to a DB and work with the data with more appropriate tools. For most cases this is true, but if you're building applications or integrations that communicate via XML, you need tooling to analyze and operate on the files that are exchanged. I get daily requests to verify feed contents, having the ability to quickly extract a specific set of items from a massive file and verify not only the contents, but the format itself is essential.
Anyhow, hopefully this can save someone a few hours, or at least get them closer to a solution.
I have a class that stores a password (I'm going to be adding more things than just the password) called Data:
import java.io.Serializable;
public class Data implements Serializable{
public String password = "";
}
As a test I ran these two:
private static File datafile = new File("data.src");
public static void checkDatafile() {
try {
Data data = new Data();
data.password = "Newwww";
if (!datafile.exists()) {
datafile.createNewFile();
ObjectOutputStream oos = new ObjectOutputStream(
new FileOutputStream(datafile));
oos.writeObject(data);
oos.flush();
oos.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
public static Data loadData(Data data) {
try {
ObjectInputStream ois = new ObjectInputStream(new FileInputStream(datafile));
data = (Data) ois.readObject();
ois.close();
} catch (IOException | ClassNotFoundException e) {
e.printStackTrace();
}
return data;
}
It writes and reads perfectly but when I open the data.src in notepad it's somewhat readable by humans and the password is not secure, this is the output of data.src:
ｬ・ sr data.Data克ﾗ淕6J・ L passwordt Ljava/lang/String;xpt Newwww
The password is easily seen and is not safe, is there a way to encrypt/encode the object when writing to a file so that it's unreadable by humans?
Also, I'd rather stick to the standard Java libs then to download and use others.
It depends on what you mean by "unreadable". If your goal is to prevent a malicious person from extracting the password, even if they have the necessary permissions to run your program, you'll be hard-pressed to do so. After all, if a program can decrypt the password, a person with the same permissions can too. Even without appropriate permissions, a malicious user could potentially inspect raw bytes in memory and extract the password if they have access to the machine.
On the other hand, if you can reasonably trust the user, but just want to avoid having them accidentally see the password in clear-text, any number of simple schemes will work; even just serializing the string as its hex codepoints would be good enough.
If you must store the password itself, i.e. you're accessing a third-party service that requires a password, you essentially have to lock down the machine and limit its access to people you absolutely trust. There's no way around that. There are a number of resources describing encrypting passwords, but they all rely on you being able to lock out users from some part of the system, generally either the encryption key or the cypher-text.
However you likely do not need to, and should not, actually be storing the password at all. The standard way to authenticate a user is to never store their password, and instead store a one-way hash of the password. This is (theoretically) impossible to decipher back into the original password, but by hashing the password the user enters when they log in and comparing it to the hash you have on file you can verify their identity without ever knowing what their password is.
Edit: One more thing about systems which need to store actual passwords. In addition to locking down the machine, you need to create a robust audit trail that records all attempts to access the passwords. Every interaction with that machine and its data should be logged and tracked so that, in the event something goes wrong, you can inspect your audit history and understand the scope of the problem. If you don't have an audit trail you'll have to assume your system is entirely compromised, because you'll have no evidence to the contrary.
not via ObjectOutputStream. you'll have to use a encryption library and either encrypt the complete file, or the password.
You can have the Data class implement the writeObject/readObject methods to encrypt/decrypt the passwords as you read and write the object.
private void writeObject(ObjectOutputStream os) throws IOException{
password = encrypt(password);
os.defaultWriteObject();
}
private void readObject(ObjectOutputStream os) throws IOException{
os.defaultReadObject();
password = decrypt(password);
}
Where encrypt/decrypt define the ecryption/decryption algorithm you wish to use. This being said and as noted in a comment by Gregor Raýman, you might consider just hashing the passwords rather than storing them.
You can try encoding/decoding your content. You can use MD5 hash-functioning. There are pre-written functions and its usage is pretty simple.
The following link can help you understand how to use it in you code.
http://www.asjava.com/core-java/java-md5-example/
I have a hashmap with large number of entry's which is serialized.If i make a small change in hashmap is it required that I overwrite the old file completely or is there an alternative ?
public class HashMapSerial {
public static void main(String[] args) {
HashMap<String,Integer> hash=new HashMap<String, Integer>(100000);
hash.put("hello",1 );
hash.put("world", 2);
//+ (100000 -2) entry's
ObjectOutputStream s=new ObjectOutputStream(new FileOutputStream(new File("hash.out")));
s.writeObject(hash); // write the hash map to file
hash.put("hello",10);
s.writeObject(hash); //rewrite the whole hashmap again
}
}
Since the change is only for the string "hello" and for no other element is it possible to update the serialized file only for the string "hello" instead of rewriting the whole hashmap once again ?
Use DB or simple File IO maintain upto where you have written previously .
AFAIK, you can't do incremental saves with the simple java serialization.
You should instead use another system to store your data (such as a database).
Maybe it's overkill but a NoSQL db (cassandra for instance) would be simpler than trying to create your own system.