I was wondering what the easiest and smartest way to organize data for easy retrieval and manipulation. I am creating a program in Java that will keep track of employee information, such as names, number, address, phone numbers, etc. The obvious solution would be to save the information in a text file, but that doesn't seem very smart or elegant. I was looking at databases, but they seem like overkill, since this information will only be accessed by one person at a time.
Databases are not overkill. They're great for organizing data even for embedded systems. (Take a look at SQLite, for example.) Other possibilities, depending on how much data you're talking about, are XML files (for which there are several APIs you could use) and Properties persisted to files.
Answering this kind of question is always like trying to answer, "what kind of car should I buy?" without more information on your needs.
Are you building a desktop application? Web application? Terminal app? What kind of information will you be storing? How much? Etc.
Databases arent anything to fear, to be sure, they can offer structured data storage in an easily retrievable and reliable format. You can also look into embedded databases like HSQL or SQLite or non-relational models like MongoDB (best if you use highly unstructured data).
But I would suggest you share a bit more information on your needs first, then we might be able to tell you if you need the SUV or the sedan.
Related
Shelve is an ultra-simple No-SQL persistence layer which allows you to trivially persist a mapping of objects. It's a commonly used package in Python because it allows you to trivially add persistence to any application.
It's simplistic nature means it's somewhat limited - but it's surprisingly useful. You can map any arbitrary hashable key onto any serializable object.
Does something like this exist for Android? I'm writing a very simple app, and I've noticed that I'm spending a lot of time faffing around with table structures, select & insert statements. That's the sort of thing I almost never do in Python since I'd usually have some kind of NoSQL alternative.
I'm not expecting to to work exactly the same way - clearly Python and Java are languages with very different characteristics. I just want something that nearly as simple to use and requires less manual SQL faffing.
One more thing - this is a fairly trivial app. I'd prefer to introduce the bare minimum of additional project dependencies. Preference will be given to solutions which require nothing more than the Android APIs.
You said preference to Java API answers so I probably won't get preference, but Couchbase Mobile is the best Android No-SQL I have ever come across.
http://www.couchbase.org/get/couchbase-mobile-for-android/current
You can use SharedPrefences from DataStore. It's pretty much what you need. You don't need full SQLite power for this.
I was thinking of building an app to serve audio content.
The first question I get is how to store it. Two obvious solutions that occur are:
Dump in database as BLOB
Dump in filesystem, and store path in DB
There was a similar question here and the answer urged to store in file-system. I can think of at least one disadvantage of storing in files, i.e. I loose all backup, recovery and other awesome features of databases.
Also I wanted to know how both solutions would fare in terms of scalability.
Does anyone know how flickr or youtube does it?
Or does anyone has even more creative(scalable :)) ideas?
Your file system should have backup and recovery procedures setup if this data is important. (The rest of the application is backed up right?). So you shouldn't use a database just for the backup and restore capability.
Storing the files outside of the database allows you to separate your database and file servers which will be a plus on the scalability side.
I would definitely go for Filesystem. storing and deliviring (large) files is exactly what it was made for.
Storing files in a file system would allow for using Content Delivery Networks. Outsource the storage may bring several benefits.
This is a classic question. And a classic argument, with good points for both solutions. Scalability can be achieved with both solutions. Distributed databases are usually easier to handle than distributed filesystems if you grow to the size where all you media dont fit on a single server (but even that is open to debate). Think MongoDB or other NoSQL scalable databases.
It boils down to what features you need. It is very hard to implement transactionality on a filesystem, so if it is a concern to you, you should use a database.
Backup and recovery of filesystem is much easier to implement than proper and consistent backup of the database. Also if you lose a file on the disk, it's just a file. If you lose a part of the huge table, it's a loss of all files contained or referenced in that table (as the table becomes unreadable).
Of course, for small databases where you can turn off the DBMS and quickly copy all DB files all of the above is not applicable, but this scenario is almost the same as having data files on the disk.
I think that both ways are viable. But the backup issue i definately there. Both solutions are scalable given the right design. But big files are probably better of in the file system.
Regards,
Morten
I want to find out more about NoSQL databases/data-stores available for use from Java, and so far I tried out Project Voldemort. Except for awfully chosen name, it seems fine so far.
I'd like to find out more about other such database systems. Now, on wikipedia article there is a list of some of them, and there is some documentation on their project pages.
However, instead of comparing technical specs and tutorials provided by authors, what I would like to know is:
What are your experiences with working with these libraries on real projects? Which one would you recommend for use based on that experience, which one you wouldn't and why?
I know that only people to be able to answer this question are those who actually used more than one such database, but I hope that someone did do so.
EDIT:
By "real project" I primarily mean a project in production (but in absence of these anything larger than a homework or finished tutorial applies).
I worked with a relational database that had enormous amount of data in it, most of it concentrated in a single table, which was denormalized for performance anyway. But, because of the entire mess with constraints etc, creating a usable cluster had shown horrible results in both stability and performance.
Now, I'm quite sure that most likely any of these NoSQL systems would be a better choice then what I had at disposal. But, there has to be a difference between them, too. Whether it is in documentation, stability between versions, community, ease of use, whatever... And there are many giants. Which ones shoulders to choose? :D
We have been working with HBase for our projects. Our experience is -
The community is very dynamic and extremely helpful
The installation procedure for developers is quite easy in either pseudo distributed or standalone mode
We have been using it for integration test like unit tests
Installing a cluster is also easy but comparing some other NoSQL it has more components to install than others.
Administering - is still going on so not able to say much to say about it.
Do not use it for SQL like SELECT queries, for that we are using Apache Solr
To make development and testing easier we have come up with a simple object mapper - https://github.com/smart-it/smart-dao
The reason I chose is HBase, like other NoSQL, solves sharding, scaling by design making it easier in the long run and that seems to hold well.
Maybe the most prominent of Java NoSQL solutions is Cassandra. It has some features beyond Voldemort (Order-Preserving Partitioner which allows range queries; BigTable style structure for values); and is missing others (no alternate storage backends or version clocks for versioning).
Its performance is more optimal for fast writes, but its biggest strength is probably ease at which it can be horizontally scaled by adding new nodes (something where V is bit more static).
Compared to, say, MongoDB, its data model is quite simple and often there's no point in using much more than key/value abstraction (that is, handle data mapping on client side, store serialized objects).
It has full replication and distribution, unlike some k/v stores (couchdb, from what I understand).
It's pretty difficult to nail down a good choice without knowing exactly what your use case is. Much of it depends on what kind of data model are you comfortable with and fits your need. You have key-value stores, document-oriented, column-oriented, etc. Another huge factor is the products take on scaling and how they choose to deal with availability/consistency trade-offs.
I like MongoDB. I like how it supports queries and I like the document oriented data models. It fits many problems that I seem to run into. There is a Great (with capital G) community as seen at the recent MongoSV event.
Your best bet it to pick 3 different products and evaluate them. I would also see if you can find some companies who have presented at conferences and tell their stories of how they were successful. Videos from MongoSV will be available soon.
I am pretty far into my first Android application, and I have the sneaking suspicion that I'm "Doing It Wrong". My app talks to a Ruby on Rails server and serializes objects back and forth via XML. Before I knew what was happening, I found myself knee deep in writing my own crappy ORM, a problem which is compounded by the fact that I haven't written any Java since high school.
My conflict here is that I want my client-side (android) app to be capable of serializing via a variety of methods, such as HTTP/XML, to a local database, or out to the local filesystem. I started out with the Strategy pattern, but I feel like my solution is badly lacking.
For one, should I re-implement all of Rails model validation on the client side, because I don't know if I'm always going to be working with Rails on the other side? The even bigger issue is that right now I can only represent flat objects as key-values, as my code can't handle nested objects like a true ORM.
I'm sure Android devs deal with this all the time, so I'm interested to hear what other people do to cope with these issues.
I wouldn't approach your Android application as an extension of a Ruby app - rather a consumer of an API. If you can try to expose your server application as JSON (or other format, but JSON is the most lightweight) and consume these APIs from your Android application you would most likely have less problems as JSON is already in K/V format.
I have not yet written Android objects to SQLite yet, but I have written them as both Parcelable objects and to the SharedPreferences. Both of these strategies are sufficient for small to mid-range apps. For data intensive apps, obviously you will have to take it a step further to SQLite.
There are some great articles for these approaches: Managing State.
It boils down to designing your objects in a way that can be serialized easily. That means no circular references or extremely complex objects. This shouldn't be a very large problem, especially if your data is in JSON format already. You simply need to extend some classes and add functions that return a Parcelable object representation or a string representation so your objects can be saved thus.
I would avoid cloning your server-side objects and validation in Android as it then requires modifying both sources if you make small changes. The server should handle all data and validation and you should simply be requesting, caching, and sending data from Android.
I'd be interested to hear if there any challenges to writing objects to SQLite, but I imagine it's not that much more of step from the details I've outlined above. Hope this helps in some capacity!
Hessian is great for RPC. You don't have to do any serialization yourself. It doesn't use XML, so it's more efficient and more appropriate for a mobile platform.
I haven't done much of persistence storage on Android but I think you need to use SQLiteDatabase and make your own Cusor that De/Serializes your object so that it can be added to the database. A possible solution would be to extend a SQLiteCursor or an AbstractCursor.
Otherwise I don't think there is other solution apart from, possibly, "hardcore" Serializabled (Which I suspect it may be too much for a phone, I may be wrong)
I think you might be going too heavy for a smart phone application. I would look at using RESTful style web services with JSON content.
Looking to your question I got the feeling that maybe you just over-complicating your requirements? Why can't you just use JSON format to represent your objects data in portable way? Then you will be able just to store it either on file system or in database in simple text field. You can leverage android-active-record library for transparent DB persistence (http://code.google.com/p/android-active-record)
I'm coming from a web-development background and I am wondering how I would make a learning algorithm in Java/C++. Not so much the algorithm part, but making the program "remember" what it learned from the previous day. I would think something like saving a file, but I suspect their might be an easier way. Apologies if this question is just over the top stupid. Thanks.
I think that would depend a bit on the problem domain. You might want to store learned "facts" or "relationships" in a DB so that they can be easily searched. If you are training a neural network, then you'd probably just dump the network state to a file. In general, I think once you have a mechanism that does the learning, the appropriate storage representation will be relatively apparent.
Maybe if you can flesh out your plan on what kind of learning you'd like to implement, people can provide more guidance on what the implementation should look like, including the state storage.
Not stupid, but a little ill-formed maybe.
What you're going to do as your program "learns" something is update the state of some data structure. If you want to retain that state, you need to persist the data structure to some external store. That means translating the data structure to some external formal that you can read back in without loss.
Java provides a straightforward way to do this via the Serializable interface; you Serialize the data by sending Serializable ojects out through an ObjectStream; the same ObjectStream will reload them later.
If you want to access and save large amounts of data maybe a database would work well. This would allow you to structure the data and look it up in an easier manner. I'm not too well versed on the subject but I think for remembering and recalling things a database would be vastly superior to a file system.
A robust/flexible solution in Java (C++ too, but I wouldn't know how) would be using a database. Java 1.6 comes with the Apache derby database which can be embedded in your apps. Using JPA (Java Persistence API) makes it easy to interface with any database you can find drivers for.
You should look into Neural Network software development. Here's a collection of nice Neural Network libraries for different languages. I am not sure if this is the easy way but once accomplished would be very handy.