I have this requirement for my business. We have a swing desktop application that works with a mysql database. At the end of each day the swing app exports the data that has changed and uploads it to a server. The set up is, a user working in an office, will have many companies that he is working with. If he changes any data for that company, then I export that company's data alone from the database. The data is exported in the form of java objects, serialised and stored into a file which gets uploaded.
The next day, if there are any changes made to that company again then I will replace the file in the server with the latest uploaded file.
Now on my server, I would like to work with this file. I would like to convert each of these files into mini databases that a webapp can read. It will not write to it. Everytime the user uploads, the database will be deleted and recreated.
So ultimately each of these files are a small subset of the data that a user has in his desktop application.
Now this issues are:
The objects that I have exported are "Apache Torque" objects. Torque is an ORM tool, basically the object represents the table. I need to convert this object into a database. Sqlite, HSQLDB, Derby...? The database should be small. If the object file is about 5KB, then the database that represents that file shouldnt be 3MB. Derby did that actually.
The java object classes could change. Since the underlying database could change. Hence I will need to deserialise these objects and create a database from it as soon as it is uploaded. Otherwise, I will not be able to deserialise these objects later on. Small changes to the database is fine for the web application. But if I dont deserialise it immediately, then I am stuck.
The conversion from the java object to the database should be fast. Since the user actually waits when his data is getting uploaded I would like to add a maximum of 5-10s additional for the conversion.
Is it ok to have thousands of these mini databases lying around? Is this design okay? Is there an alternate solution?
I wouldn't try to put each dataset into its own database. I would put all of them in one big database, along with a column in the key tables indicating the dataset that each row applies to (this sounds like it should just be a company identifier). This is a more normalised design than having many small databases.
You will then need to write the webapp so it makes queries for particular datasets, rather than connecting to a particular database.
if you adopt that approach, you can deserialize and store the datasets as soon as they arrive. The storage is simply inserting rows into an existing database, so it should be very fast.
In addition, i expect that one big database will be much easier to manage, maintain, report on, etc, than many small databases.
If you tell us more about the details of your schema, we could discuss how the database could be organised, if that would be useful.
Related
I am going to make a business application for my father to make GST(Goods and Services Tax) filing easier. I have the design ready and I am going to use JavaFX.
The user will enter the data in tableview and that data needs to be stored for future reference.
The tableview needs to be converted to an excel file (gonna use Apache POI). The excel file will be sent to a C.A who will file GST on my father's behalf.
The application will need to import/export data into/from the tableview and edit the data as necessary.
I have 2 options :
Store/retrieve data from MySQL to tableview, update it according to the user's will and later export the data into excel files for sending it to C.A.
Store/retrieve data from excel files to tableview, update it according to the user's will and send the excel file to C.A.
I am planning to expand the application into a complete Business software that can manage entire business.
What should I use?
Which one will be more efficient and why?
I hope I am able to convey my question (I ain't good at writing).
In my own opinion it is more efficient and have more posibilities to explotes the data using MySQL, because reading and writing an Excel file will take a lot of time and it is slower.
I'll answer my own question, since I have got the answer.
I'm going with SQLite for now as using csv or excel files is gonna consume a lot of resources (I tried it).
I am going to sync the .db file in drive using scripts from the application itself. MySQL is definitely better choice but I want to database to be used by 2 computers at a time (not in network) so I will have to pay for online database.
I will store the .db file and drive and will retrieve it whenever the application runs. In this way its going to be safe.
I cannot understand how distributing Java programs that use a database works.
Let's say I am using Derby as RDBMS and I want to store tasks and calendar entries in a database.
I want each user of the program to have a local database.
But I don't understand how in-memory databases are supposed to work. Should I write a script so that the first time my program is launched it creates a database and empty tables? Or will they be already created during the installation of the program?
If your program wants to store the user's tasks and calendar entries in a database, you probably don't want to use an in-memory database, because the in-memory database disappears when your program exits.
Rather, you want to use an ordinary persistent Derby database, which will store the user's data in files in a folder on the filesystem.
You do indeed have to create the database and issue the CREATE TABLE etc. statements to create the tables in that database. You could provide that as a separate script, or you could have your program issue those statements itself.
Tables are not automatically created, though; you have to issue the CREATE TABLE statements one way or another.
This may be a dumb question, but I have a project for a class where I have to store/retrieve files from a SQL database that connects to a web page. Now, I could just make a webpage to store pictures or music files but I am currently working on creating some basic games in java. I know that there are ways to be able to access these files from a web page, but like I said, the project has to include a SQL database.
So my question is, is there any way to store and retrieve these kinds of files from the SQL database? Being able to download the files would be fine as long as the user would be able to open them, though I would prefer the user be able to open them in the browser.
If anybody has any suggestions I would appreciate it.
When storing into a SQL database, you don't really store the files. You store the file contents. In it's most generic form, you could make a table with a big binary field (a blob or clob depending on which database you use) or a big text field (a varchar) and put the contents of the file into that. Other columns could store file names and such.
To really leverage the SQL database, you would want to know enough about the content of the files to take advantage of indexing and such by breaking it up into more detailed parts. For example, if you are putting a save file in there, you could make a detailed table with columns for username, and all sorts of game-specific state that needs to be saved.
I am collecting data from a website and trying to save it to a database (or something similar that is very accessible) rather than having a heap of files on my desktop or in a folder.
There are many pages that I need to look at (1900 to be exact). I want to save time in getting this data, and decided to make a Java program to do this.
This is basically what I am trying to do.
Visit the webpage: www.TestWebsite.com/items/0
Save the (Name, Description, Image(png)) into one array/class to a Database.
Repeat until I get up to: www.TestWebsite.com/items/1899
I want to be able to access this data offline without having to need to go online to view it.
Any ideas on how I should start. I have made a basic webpage viewer, I am just missing the step in between saving the strings and images to a database.
I appreciate any help!
Actually just did this the other day. I used jsoup to scrap the webpages I needed and wrote to my local database. awesomely easy framework for webpage parsing.
It's fairly straightforward, but you'll need to learn a little SQL if you haven't already.
You'll also have to pick a database platform - I'd suggest SQLite for such a purpose, since the data is for personal use and it's lightweight and easy to set up.
Here's a tutorial on using JDBC (Java Database Connectivity) to talk with a SQLite database: http://en.wikibooks.org/wiki/Java_JDBC_using_SQLite. It goes from setup to inserting data, so once you've completed that it should be straightforward to modify your webpage viewing code to grab the data you need and shove it into the DB.
Good luck!
I want to save and load database on disk.
What I really want is to be able to do the typical application save as and open things with the database. Means when I want to save the database, I will click the save as button, and give a name to the database, and then save it. Later I want to be able to load back the database, by using open button to find the path to the database.
I'm using sqlite and java, and I heard that firefox bookmark manager using sqlite to store bookmarked data. And I don't know the correct term but roughly I want to be able to do like firefox bookmark manager to save and load the database.
Hope you guys can shed some light here.
You can use SQLite database files in two possible ways:
Like a database: Upon the first run of your application, you create the SQLite database file and create the schema (using CREATE TABLE SQL commands etc.). Then, whenever you want to change the saved data, you access your database file and execute single UPDATE, INSERT or DELETE statements to modify exactly those records that have changed. This makes operations such as Save as not quite straightforward, but it's comparably fast for large amounts of data where only small parts are modified.
Like a data file: Every time you save your data, you create a new database file (if there was one with the same name before, you delete it first). You then create the whole schema, and then you write all the information to the database file (using INSERT SQL statements). This allows you to handle things with the traditional Save/*Save as* commands.
For more detailed information, please ask more specifically; in particular, outline your problem if you need to know which approach serves you best.