I need some suggestions for the App am developing,
I have some 6 sensors which are continuously sending the data to the App, I need to log the data in some file. I want to know which is the better approach to store such data. I tired using json by storing all the values in one json objects, and created a array of json objects somewhat like this
[{sensor1, senor2, sensor3, sensor4, sensor5, sensor6},{sensor1, senor2, sensor3, sensor4, sensor5, sensor6}]
But I have more then 20000 data like this to store, creating the array of 20000 json object and writing to file in one shot looks expensive. on that data is coming once in 15 seconds. writing one JSON object once in every 15 second will disturb the JSON format.
So I need suggestion like is using JSON method to store this data is better or I should think of some other method like CSV??
If you want to store the data points over time, have you considered using a time series database? There are various NoSQL style databases that are well suited to this task, Cassandra being one
https://academy.datastax.com/demos/getting-started-time-series-data-modeling
Without a doubt, CSV.
But only if you're certain your format will not change (always 6 sensors). Furthermore, in CSV, data can be added at the end of a file while in JSON you have to write in the middle.
Related
I am relatively new to Java and have much more experience with Matlab. I was wondering what the best way is to store a relatively small amount of data, which has been calculated in one program, that should be used in another program.
Example: program A computes 100 values to be stored in an array. Now I would like to access this array in program B, as it needs these values. Of course, I could just write one program all together, which also implements the part of A. However, now every time I want to execute the total program, all the values have to be calculated again (in part A), which is a waste of resources. In Matlab, I was able to easily save the array in a .mat file and load it in a different script.
Looking around to find my answer I found the option of serializing (What is object serialization? ), which I think would be a suitable for doing what I want. My question: is serializing the easiest and quickest solution to store a small amount of data in Java, or is there a quicker, more user-friendly option (like .mat files in Matlab)?
I think you have several options to do this job. Java object serialization is one possible way. From my point of view there are other options to serialize the data:
Write and read a simple text file to store the computed values.
Using Java Architecture for XML Binding (JAXB) to write annotated Java classes to XML file. Same for JSON is also available.
Using a lightweight database like SQLite or HSQLDB (native Java database).
Using Apache Thrift or Protocol Buffer to de/serializing Java objects to files.
My question is basically the title: I have a number of large numpy arrays that I want to port to a Java application, eventually. The only way I see myself doing this is by first transferring this data to Jython. However, I am not sure how to do this as numpy doesn't exist in Jython.
Well, Python will easily let you serialize your data to files in whatever format you want in 3 lines of code. What format your java application can read from?
If you don't want to write data to disk, or even can't duplicate the in-memory data to pass to other process one thing to check is cap'n'proto -https://capnproto.org/
One way of serializing the arrays as json encoded data files is simply:
import json
json.dump(myarray.tolist(), open("myfile.json", "wt"))
If you Java side can read json, that is all you need.
In my software I want to store some data, that later they will be used. Something like a database to hold data:
Date, source path, destination path, and an array of file names.
Also another table to hold information about ftp connection:
Host, port, username and password
I need to know what methods are available to store and parse these data. I noticed there is a file type called .csv, is this an option for me? And is there any other option?
I think this depends a lot on how much data you want to store and how you need to access it.
If your application is going to be collecting a lot of structured data, such as user profiles, or product information, ie, if your application is all about a database then, yes as others have commented some sort of SQL database would make sense.
If your needs are more along the lines of just storing some "session" information, maybe like the last state of a GUI form for example, you might want to just serialize the data and write it to a simple text file.
One simple way to do that would be to serialize the data in a human readable format such as JSON and then write the text to a file, and then read it back and deserialize it when you need to restore it from storage.
If this is what you are looking for take a look at gson (from google), it provides a very easy what to convert a java object to JSON and back again.
JSON, is just text, so you can just read and write it to a simple text file.
In my program, I am reading a series of text files from the disk. With each text file, I process out some data and store the results as JSON on the disk. In this design, each file has its own JSON file. In addition to this, I also store some of the data in a separate JSON file, which stores relevant data from multiple files. My problem is that the shared JSON grows larger and larger with every file parsed, and eventually uses too much memory. I am on a 32-bit machine and have 4 GB of RAM, and cannot increase the memory size of the Java VM anymore.
Another constraint to consider is that I often refer back to the old JSON. For instance, say I pull out ObjX from FileY. In pseudo code, the following happens (using Jackson for JSON serialization/deserialization):
// In the main method.
FileYJSON = parse(FileY);
ObjX = FileYJSON.get(some_key);
sharedJSON.add(ObjX);
// In sharedJSON object
List objList;
function add(obj)
if (!objList.contains(obj))
objList.add(obj);
The only thing I can think to do is use streaming JSON, but the problem is that I frequently need to access the JSON that came before, so I don't know that stream will work. Also my data types on not only strings, which prevents me from using Jackson's streaming capabilities (I believes). Does anyone know of a good solution?
If you're getting to the point where your data structures are so large that you're running out of memory, you'll have to start using something else. I would recommend that you use a database, which will significantly speed up data retrieval and storage. It will also make the limit of your data structure the size of your hard drive, instead of the size of your RAM.
Try this page for an introduction to Java and Databases.
I can't believe that you really need nearly 4GB RAM only for text files and JSON.
I see three possible solutions.
Switch to plain text if it's possible. That is not that memory hungry.
Just open and close the files as you need them. You can order the files to a specific naming convention, like the first two/three/... digits of their hashes, and open them as you need them.
If you have so many data, you could maybe switch to a database. That would save a lot of resources.
I would prefer option 3 if it's possible for you.
you can make api and get responce.body from it
I have an application which stores information in a JList. However, of course, when the application is closed all of the information is deleted from memory.
I'm trying to build the app so that when re-launched, it will contain the same data. So is there a way to store this data in a database or similar and if so? Where and how do I go about this?
The simplest way to persist IMHO is in a File.
Try using Properties if you need a key-value map.
Or, if it you're binding more complex objects I recommend a Simple XML serialization package.
You need to connect your application to a database using JDBC. JDBC stands for Java Database Connectivity. As you can see from the name, it lets you to connect to a database. Hence, you can link your application to a database,and store your data persistenly.Here's a link to start off with. And here is something for further reading.
If the data is not complex and is not large (more than a few instances of a few objects) you could persist the list to a file using serialization. This will get you started. If you list is large or complex you might consider a database. Searching for JDBC will in your favorite search engine will get you started.
I think you want a plain flat file. It's simple; you can have one going in no time. (The learning curve is much less than with databases.) And it's fast; you can read a 1 GB file before you can even log on to a DB. Java serialization is a bit tricky, but it can be a very powerful way to save vast amounts of complicated data. (See here for things to watch out for, plus more helpful links.) If, for instance, you wanted to save a large, complex game between sessions, serializing it is the way to go. No need to convert an Object Oriented structure to a relational one.
Use a database:
if you want to add data to a large file, or read only part of the data from a large file. Or if other processes are going to read and modify it.
Consider a DB:
if you are already using one for other purposes. If the user might start on another machine and have trouble finding the file from the last session and the data is not too extensive. Or if the data is relational in nature anyway and someone else may be interested in looking at it.
So if you have a simple case where the user always starts in the same directory, just write and read a simple file. If you have a lot of complex, extensive OO data, use a flat file even if it is not easy to do--you'll need the speed. Otherwise, think about a DB.