We Keep Coding

Passing images to Java

I am working on a project where I have extracted images from sensor and saved them to the operating system directory. I have a Java API for uploading images to the server.
I need to upload these images and some other data typically float data type to the main server.
I need to decide an inter-mediator such as a database where I store those images and make connection through java to upload them or use HDFS.
Can some body please advise me, which option will be best for storing images? Database or HDFS?
Note: Images are up to 150 thousand can be more in future.

I think the best way to do that is to keep the floats you need and metadata of the images in the database. For easier searching and querying and easier interaction with the Java. The actual images are best stored on a file system to decrease the transformation from and to the database. I believe a simple file system would be good enough for that size of images. You probably won't use any of the fancy HDFS functions like map reduce and stuff like that. But that's up to you.
So in this case if a standard file system isn't good enough for you and you want something bigger then HDFS is the way to go. So the proper way would be a mixture of the two.

It totally depends on the usecase , you can choose
HDFS : when you wanna read them as a whole or transfer or process them to do any manipulation upon the images data and store or do someother action based on the processed results. In simple, if you wanna do Map-Reduce operation. And reading images in HDFS is sequentially , if you wanna perform to fetch particular image based on certain selection criteria, then it costly and performance impacted operations.
Database : It is better for query based operation where you wanna query or do DML operations upon images on certain criteria basis, In simple, WHERE conditions. But this is totally time consuming process, when you wanna process as a chunk. And the performance will be obviously very slow as you wanna store 150thousand of images
So My suggestion based on the requirement, you wanna store images as intermediate, it will be better to store in HDFS itself.

150.000 images is not considered a huge amount today. If an average of 10 MB is assumed for each image (uncompressed) the amount of data is 1.5 TB, which should be possible to store in an off-the-shelf database (with off-the-shelf hardware, i.e. a Linux box with some RAID disks) like postgreSQL. I'm no expert in HDFS even though I tried products in the same family as HDFS I find them easy to use, I guess you could try Hadoop then for processing of the images as well if you are looking for a way to parallelize the processing. Even though this product family is nice I would still use a standard database like postgreSQL if parallelisation is not really needed by nature (like you get in HDFS).

How do I store data in Java without need of extra software?

I am currently developing a program in Java using NetBeans that manages expenses. In this program I used MySQL to store data. Now I want to ask that if I give the program to one of my friend, he would also have to install MySQL using the same password that I used. Is there a way in which he will not be required to install MySQL?
Now suppose if my friend already has MySQL, but with a different password. My program would not work in that case, and it would be hard to replace my password with his password in the code. Is there a way to make the program run on his PC?
Earlier once, I have used an Oxford dictionary program. That time I did not have Microsoft Access installed. When I installed Microsoft Access I came to know that all the words of the dictionary are stored in a Microsoft Access file. How can I do this? I.e. how can I use MySQL/Microsoft Access to store data without the need to install either of them?

You can use an in-memory database like H2 Database if you don't require a large amount of data
But I think you should make your db connection configurable by using a properties file

If you want everyone to be able to use the database, you need to run it on a server that people can access through the internet.
However if you don't care about them using the same database and just want them to use their own, you could for example create a small file named "config.ini" or something like that and put the login information (like the password) in there.
Then just read it in your code and use the info you read to log into your database.
That way, every new user will only have to change the config.ini file without ever having to touch the code.

The best solution in my opinion would be SQLite as it is light, and data can be stored locally in a desired location in a single file. H2 is more likely to be a developer tool.
This solution does not require additional software to be present on the user machine. Of course it has its limits, but as a simply storage for program dynamic data it is a good solution. It is worth mentioning that Android applications also can store their data in LiteSQL. It is a bit different approach there, but the principle stays the same.

Attachments under File system vs databases?

I need to sore attachments at server side. I can store them either under blob column of database or under file directory.
My question is which one is more reliable, scalable and maintainable?
EDIT:-
if we go for file system, we have to handle synchroniztion yourself. Is n't it ? For example if two users are trying to create/update the File under same directory how will we handle concurrency with filesystem?

Storing data in directory is more reliable due to indexing and data fetch and other operation. Just store the path of the file into DB and store that file into directory.
When there's lot's of data store request came on server it's very hard and complex to handle so much request.
So it's better to store data on directory so accessing of data becomes more faster and when the daily scale of DB storage increase then these become so important so when you start any system first of all study it well and then decide that what to do or which technique will be the best ?
When more data are there in DB then clustering and indexing become more important.
If you want to use it for small data storage then blob it good option but for large data I ll not recommend you because I have made online data store web application and faced this situation so at end I have used to store data in directory and just path in DB.

In a Spring MVC Hibernate application, where should I save user images?

I am creating a Spring MVC Hibernate application using MySQL. Where should I save the User Images: in the database or in some folder, like under WEB-INF/ ?

Certainly not inside WEB-INF. You might want to save them in the file system, but not in the webapp's directory. First of all, it could very well be inexistent, if the app is packaged as a war. And second, you would lose evrything as soon as you redeploy the app. Desktop apps don't store their user data in their install directory, do they? It should be the same for webapps.
Now, since images are usually big, and they're not searchable, you might want to store them on the file system, and only store their name, path, hash, and/or mime type into the database. But it depends on your application, if they need to be served/used by other applications, if these apps have access to the database and/or the file system, etc. You decide.

You can choose it:
DataBase - you have the positive point that this can be associated with records and will never be orphan (depending on your model). For backup it is a little bit painful for situations in which the database increases.
FileSystem - backup facility, as these are physical files, an rsynch process should be enough. Another positive point is that you reduce the IO from DB. Although, it is quite hard to attach a logic between the files and the record stored in the DB (you have things distributed), so you will not be sure if the file wasn't removed and there are still some records referring to it in DB.
If filesystem option is chosen, put it outside the application directory structure (prepare a dedicated place for the files). The application dir should not be modified, causing some pain when redeployment is done. You can use symbolic links though.
With images, probably you want to perform some thumbnails and so on, this would be cheaper using FileSystem option.

That depends what you're trying to accomplish.
If these are static images , and you have a fixed number of users ,you can consider saving them under WEB-INF/.
However, most likely this is not you case, and you have varying number amount of users and you have to store a user for each one of them.
Possible solutions:
A. For the user store an image name, and have a convention of storing/loading from a well known directory in your file system.
B. Store the image as a blob in your DB. Consider checking the #Lob annotation

Storing large number of images database or filesystem

I am working on a project where we need to store large no of images say some 10 millions so which is the best way to store the images.Best way in terms of speed and efficient.
It is a web based project so the image retrieval should be fast.
Database
Storing images as base64 in database.
we are working on a nosql database.
File System
To make an unique id and store it under an folder.

1)Database
will require much code for processing image as using streams
Heavier load on the database server
database storage is usually more expensive than file system storage
databases win out where transactional integrity between the image and metadata are important.
it is more complex to manage integrity between db metadata and file system data
it is difficult (within the context of a web application) to guarantee data has been flushed to disk on the filesystem
2) File system
To store images on a unique id and storing it to harddisk will be a better option .
things like web servers, etc, need no special coding or processing to access images in the file system
refer http://perspectives.mvdirona.com/2008/06/30/FacebookNeedleInAHaystackEfficientStorageOfBillionsOfPhotos.aspx
also see Storing Images in DB - Yea or Nay?

There is a trade off - it will depend on your exact situation and needs. The benefits of each include
Filesystem
Performance, especially caching and I/O
Storing file paths in the database to be best.
There are a couple of issues:
database storage is usually more expensive than file system storage
you can super-accelerate file system access with standard off the shelf products
for example, many web servers use the operating system's sendfile() system call to asynchronously send a file directly from the file system to the network interface. Images stored in a database don't benefit from this optimization.
things like web servers, etc, need no special coding or processing to access images in the file system
databases win out where transactional integrity between the image and metadata are important.
it is more complex to manage integrity between db metadata and file system data
it is difficult (within the context of a web application) to guarantee data has been flushed to disk on the filesystem
Database
Easier to scale out to multiple web servers
Easier to administer (backup, security etc)
If you have a SQL 2008 DB, have a look at FileStream in this SO article - this gives the best of both worlds.
See Storing Images in DB - Yea or Nay?
Edit
See for Nosql:
Is it a good idea to store hundreds of millions small images to a key/value store or other nosql database?
Storing images in NoSQL stores