I am inexperienced with Spring and I've been reading up on persistence options in Spring, as I am trying to find a suitable way to store data without the use of a database such as Oracle or MySQL etc...
When my app loads, it will read a file containing IDs. As the app runs, it may gain new IDs which will need to be written to the file in case of a crash. From what I can tell, I will need to replace the whole file each time, which is fine, as the data should be held in RAM and I can just overwrite the original file.
What I would prefer, however, is a way in Spring, or even Java, to sync the file and the data so that if I add 1 new ID to my list, it would automatically add a single line to the end of the file without me needing to write additional file management code. I know I can probably just concatenate the line, but something that basic probably won't be thread safe, and thread safety is a major concern here. I'd rather find a ready-made lib rather than re-invent the wheel.
So, can anyone point me in the direction of a tutorial, or technology, that allows for what I need? Or tell me if one exists, or how best I should go about this?
Thanks.
EDIT: It seems Springs resource bundle is the way forward. But I don't think it does exactly what I need to do. Using this, I will have to write code to both add to the map, and then add to the file.
Take a look of SQLite
Is a thread safe and server less sql database with Java driver.
EDIT
Other option is spring batch support for flat files.
see http://docs.spring.io/spring-batch/reference/html/readersAndWriters.html#flatfiles
Related
I am facing a problem for which I don't have a clean solution. I am writing a Java application and the application stores certain data in a limited set of files. We are not using any database, just plain files. Due to some user-triggered action, certain files needs to be changed. I need this to be a all-or-nothing operation. That is, either all files are updated, or none of them. It is disastrous if for example 2 of the 5 files are changed, while the other 3 are not due to some IOException.
What is the best strategy to accomplish this?
Is embedding an in-memory database, such as hsqldb, a good reason to get this kind of atomicity/transactional behavior?
Thanks a lot!
A safe approach IMO is:
Backup
Maintain a list of processed files
On exception, restore the ones that have been processed with the backed up one.
It depends on how heavy it is going to be and the limits for time and such.
What is the best strategy to accomplish this? Is embedding an in-memory database, such as hsqldb, a good reason to get this kind of atomicity/transactional behavior?
Yes. If you want transactional behavior, use a well-tested system that was designed with that in mind instead of trying to roll your own on top of an unreliable substrate.
File systems do not, in general, support transactions involving multiple files.
Non-Windows file-systems and NTFS tend to have the property that you can do atomic file replacement, so if you can't use a database and
all of the files are under one reasonably small directory
which your application owns and
which is stored on one physical drive:
then you could do the following:
Copy the directory contents using hard-links as appropriate.
Modify the 5 files.
Atomically swap the modified copy of the directory with the original
Ive used the apache commons transactions library for atomic file operations with success. This allows you to modify files transactionally and potentially roll back on failures.
Here's a link: http://commons.apache.org/transaction/
My approach would be to use a lock, in your java code. So only one process could write some file at each time. I'm assuming your application is the only which writes the files.
If even so some write problem occurs to "rollback" your files you need to save a copy of files like upper suggested.
Can't you lock all the files and only write to them once all files have been locked?
I'd like to save persistent objects to the file system using Hibernate without the need for a SQL database.
Is this possible?
Hibernate works on top of JDBC, so all you need is a JDBC driver and a matching Hibernate dialect.
However, JDBC is basically an abstraction of SQL, so whatever you use is going to look, walk and quack like an SQL database - you might as well use one and spare yourself a lot of headaches. Besides, any such solution is going to be comparable in size and complexity to lighweight Java DBs like Derby.
Of course if you don't insist absolutely on using Hibernate, there are many other options.
It appears that it might technically be possible if you use a JDBC plaintext driver; however I haven't seen any opensource ones which provide write access; the one I found on sourceforge is read-only.
You already have an entity model, I suppose you do not want to lose this nor the relationships contained within it. An entity model is directed to be translated to a relational database.
Hibernate and any other JPA provider (EclipseLink) translate this entity model to SQL. They use a JDBC driver to provide a connection to an SQL database. This, you need to keep as well.
The correct question to ask is: does anybody know an embedded Java SQL database, one that you can start from within Java? There are plenty of those, mentioned in this topic:
HyperSQL: stores the result in an SQL clear-text file, readily imported into any other database
H2: uses binary files, low JAR file size
Derby: uses binary files
Ashpool: stores data in an XML-structured file
I have used HyperSQL on one project for small data, and Apache Derby for a project with huge databases (2Gb and more). Apache Derby performs better on these huge databases.
I don't know exactaly your need, but maybe it's one of below:
1 - If your need is just run away from SQL, you can use a NoSQL database.
Hibernate suports it through Hibernate OGM ( http://www.hibernate.org/subprojects/ogm ).
There are some DBs like Cassandra, MongoDB, CouchDB, Hadoop... You have some suggestions Here
.
2 - Now, if you want not to use a database server (with a service process running always), you can use Apache Derby. It's a DB just like any other SQL, but no need of a server. It uses a singular file to keep data. You can easily transport all database with your program.
Take a look: http://db.apache.org/derby/
3 - If you really want some text plain file, you can do like Michael Borgwardt said. But I don't know if Hibernate would be a good idea in this case.
Both H2 and HyperSQL support embedded mode (running inside your JVM instead of in a separate server) and saving to local file(s); these are still SQL databases, but with Hibernate there's not many other options.
Well, since the question is still opened and the OP said he's opened to new approaches/suggestions, here's mine (a little late but ok).
Do you know Prevayler? It's a Java Prevalence implementation which keep all of your business objects in RAM and mantain Snapshots/Changelogs in the File System, this way it's extremely fast and reliable, since if there's any crash, it'll restore it's last state and reapply every change to it.
Also, it's really easy to setup and run in your app.
Ofcourse this is possible, You can simply use file io features of Java, following steps are required:-
Create a File Object
2.Create an object of FileInputStream (though there are ways which use other Classes)
Wrap this object in a Buffer object or simply inside a java.util.Scanner.
use specific write functions of the object created in previous step.
Note that your object must implement Serializable interface. See following link,
I am developing an application that will make heavy use of .sql files. While I am just at the beginning of development, I want to make sure I am going in the right direction to avoid re-coding later. Much like java source code is not meant for the end user to be seen, neither are .sql files and commands. My main goal is to hide them from the end user. My approach is as follows, and I am seeking alternative approaches and suggestions:
Write a small program that loops through all .sql files in a directory and stores the contents into a java.util.map using bufferedInputStream. The Map will be constructed with = new HashMap(, ). (I believe this is correct syntax). The key will be the .sql file name and value will be the .sql file contents. Then, serialize the Map object into a single file (say "SQLBin.bin") using ObjectOutputStream. Place the SQLBin.bin file into the resources folder of the main project and then use .getResourceAsStream() to access it and recreate a Map object in the main application. This will then allow me to access the SQL commands by simply referring to the .sql file by name in the Map object.
PS: I am relatively new to java. So please be extra clear.
You have an interesting technique; however, it is not clear what problem you intend to solve.
If you want to hide the SQL from the end user, you don't need to do all of this; just embed the SQL into a statement "string array" in a class and be done with it. But even such a solution might not be desirable depending on the true problem you are attempting to solve.
Also, some places really want to look at your SQL, because a database server isn't like a JVM. Your SQL can impact the correct operation of other mission critical programs. SQL servers may require manual configuration to grant you access. SQL servers may be monitored 24 / 7 for conditions that lead to excessive memory consumption or excessive use of CPU cycles. Professionals might rewrite your statements or tune the server to better accommodate unforeseen issues.
With a JVM the resource is less of a shared resource, and thus there's less potential for damage, in the worst case, you kill the offending JVM process, which rarely impacts the other processes that weren't explicitly written to integrate with yours.
I have a program where I need to save a running application to be able to go back to it later
I know that I can write/read from a text file to achieve this but the program is pretty prodigious so it's not really a good way to do it because I have 10+ classes and thousands of JTextFields, JComboBoxs, etc. Does anyone know of a way I can achieve this without writing/reading from text files?
An example of what I need to be able to do is this:
In Microsoft Excel you can load files (.exl) into it and be able to edit them.
The Swing Application Framework provides a way to save session state when your application exits and restore the state when you restart. Session state is the graphical window configuration of your application. This state includes window size, internal frame locations, selected tabs, column widths, and other graphical properties.
How do you think Excel does this? It stores the type and value of each cell, along with metadata describing the worksheet in its own proprietary binary format in a file. If you have a custom application with complex internal state, you will have to design a storage format and serialize the state yourself. You may be able to use Java Serialization, but not without some effort.
A good way to do this is to save the data from your controls into a canonical form and then make that class serializable. You can then persist that data to a file. Here's a link about serialization in Java.
UPDATE
I just noticed that you said you have thousands of form controls. So you probably don't want to do all of this in one class, but you probably want to maintain a hierarchy of classes and split out the data into separate classes. This will also help you separate your concerns. Hopefully you have POJOs or domain classes that represents your data. If that is the case, your task will be much easier. This is also why separating concerns is good :).
To save the state of an application, i can think of two popular way:
1) save the state of the application in a Database
2) save the state of the application in a binary file or XML,json or any format you want.
Maybe giving more details about the app. would help.
is it a Web app, fat client app, client/server app... !^
Solution may vary with the type of application.
Hope it help.
I am creating a few JAX-WS endpoints, for which I want to save the received and sent messages for later inspection. To do this, I am planning to save the messages (XML files) into filesystem, in some sensible hierarchy. There will be hundreds, even thousands of files per day. I also need to store metadata for each file.
I am considering to put the metadata (just a couple of fields) into database table, but the XML file content itself into files in a filesystem in order not to bloat the database with content data (that is seldomly read).
Is there some simple library that helps me in saving, loading, deleting etc. the files? It's not that tricky to implement it myself, but I wonder if there are existing solutions? Just a simple library that already provides easy access to filesystem (preferrably over different operating systems).
Or do I even need that, should I just go with raw/custom Java?
Is there some simple library that
helps me in saving, loading, deleting
etc. the files? It's not that tricky
to implement it myself, but I wonder
if there are existing solutions? Just
a simple library that already provides
easy access to filesystem (preferrably
over different operating systems).
Java API
Well, if what you need to do is really simple, you should be able to achieve your goal with java.io.File (delete, check existence, read, write, etc.) and a few stream manipulations with FileInputStream and FileOutputStream.
You can also throw in Apache commons-io and its handy FileUtils for a few more utility functions.
Java is independent of the OS. You just need to make sure you use File.pathSeparator, or use the constructor File(File parent, String child) so that you don't need to explicitly mention the separator.
The Java file API is relatively high-level to abstract the differences of the many OS. Most of the time it's sufficient. It has some shortcomings only if you need some relatively OS-specific feature which is not in the API, e.g. check the physical size of a file on the disk (not the the logical size), security rights on *nix, free space/quota of the hard drive, etc.
Most OS have an internal buffer for file writing/reading. Using FileOutputStream.write and FileOutputStream.flush ensure the data have been sent to the OS, but not necessary written on the disk. The Java API support also this low-level integration to manage these buffering issue (example here) for system such as database.
Also both file and directory are abstracted with File and you need to check with isDirectory. This can be confusing, for instance if you have one file x, and one directory /x (I don't remember exactly how to handle this issue, but there is a way).
Web service
The web service can use either xs:base64Binary to pass the data, or use MTOM (Message Transmission Optimization Mechanism) if files are large.
Transactions
Note that the database is transactional and the file system not. So you might have to add a few checks if operations fails and are re-tried.
You could go with a complicated design involving some form of distributed transaction (see this answer), or try to go with a simpler design that provides the level of robustness that you need. A possible design could be:
Update. If the user wants to overwrite a file, you actually create a new one. The level of indirection between the logical file name and the physical file is stored in database. This way you never overwrite a physical file once written, to ensure rollback is consistent.
Create. Same story when user want to create a file
Delete. If the user want to delete a file, you do it only in database first. A periodic job polls the file system to identify files which are not listed in database, and removes them. This two-phase deletes ensures that the delete operation can be rolled back.
This is not as robust as writting BLOB in real transactional database, but provide some robustness. You could otherwise have a look at commons-transaction, but I feel like the project is dead (2007).
There is DataNucleus, a Java persistence provider. It is little too heavy for this case, but it supports JPA and JDO java standards with different datastores (RDBMS, object storage, XML, JSON, Excel, etc.). If the product is already using JPA or JDO, it might be worth considering using NataNucleus, as saving data into different datastores should be transparent. I suppose DataNucleus supports splitting the data into several files, creating the sensible directory/file structure I wanted (in my question), but this is just a guess.
Support for XML and JSON seems to be experimental.