I have been working on an application using GAE in eclipse and I have a bunch of data objects. Sometimes I need to change their type, ie String -> Text so they can store more data.
What is the quickest easiest way to do a bulk update on the data/object store? I know I could probably write Java code to iterate over each object, but surely there is an easier way?
Normally there is no other way than iterating the store and changing the data by hand. The datastore is not otherwise accessible. However starting from version 1.3.3 of the SDK there is now the possibility to use SQLite as the datastore backend. To enable, set the flag --use_sqlite=true
You'll need then to find the SQLite file and should be able to use any SQLite client to manipulate the data.
UPDATE: As Nick Johnson noted, SQLite support is only available for the Python SDK and the data is encoded, making the task of directly editing the tables content rather difficult depending on the change. This invalidates this answer given the poster is looking for an easy, Java based solution :/
The best option for this is the newly released appengine-mapreduce library, which has both Java and Python versions.
In the case of converting from String to Text, though, there's no need to go through and update old entities manually - they'll be fixed when they're next written by your app, and will still work correctly in the meantime.
Related
I am inexperienced with Spring and I've been reading up on persistence options in Spring, as I am trying to find a suitable way to store data without the use of a database such as Oracle or MySQL etc...
When my app loads, it will read a file containing IDs. As the app runs, it may gain new IDs which will need to be written to the file in case of a crash. From what I can tell, I will need to replace the whole file each time, which is fine, as the data should be held in RAM and I can just overwrite the original file.
What I would prefer, however, is a way in Spring, or even Java, to sync the file and the data so that if I add 1 new ID to my list, it would automatically add a single line to the end of the file without me needing to write additional file management code. I know I can probably just concatenate the line, but something that basic probably won't be thread safe, and thread safety is a major concern here. I'd rather find a ready-made lib rather than re-invent the wheel.
So, can anyone point me in the direction of a tutorial, or technology, that allows for what I need? Or tell me if one exists, or how best I should go about this?
Thanks.
EDIT: It seems Springs resource bundle is the way forward. But I don't think it does exactly what I need to do. Using this, I will have to write code to both add to the map, and then add to the file.
Take a look of SQLite
Is a thread safe and server less sql database with Java driver.
EDIT
Other option is spring batch support for flat files.
see http://docs.spring.io/spring-batch/reference/html/readersAndWriters.html#flatfiles
I'm new to the SQLite database, and more generally, to the concept of embedded databases altogether. I'm used to creating a connection string and connecting to a remote DB server (MySQL, MSSQL Srv, Oracle, etc.). I know this question is probably quite silly, but being in uncharted waters here, I can't seem to find the answer to this on my own.
So I'm writing a Java app that uses SQLiteJDBC as the Java driver for SQLite (the app's embedded db) and am creating the tables and inserting records into them from the Java app itself. What I'd like to do is download/install SQLite on my system - completely independent of the Java app - and then write SQL scripts that will do the "skeletonizing" (creating & insertions) of the database file itself, then copy that .sqlite file into my project directory where the app can then use it.
I'm just finding it incredibly difficult to develop database schema from inside the Java app itself; just seems like an unnecessary step.
So, my question:
Is this even possible? To create, say, myProgramDB.sqlite off the command line with the SQLite tool, and then (essentially) cut-n'-paste that file into my Eclipse/NetBeans project (of course, in the right directory!) and have it work? This is also assuming I have correctly imported the SQLiteJDBC JAR into my project through the IDE. I just want to create the DB somewhere else, then copy it into my project, instead of developing the DB through my app directly.
Thanks for any insight!
Just think of the database as a normal file which your app refers to either by an absolute or relative file path, so with that in mind embed it in your project like you would any other file in Eclipse (or point to a specific location where you expect it to be).
If you're going to create your db manually, SQLiteStudio (http://sqlitestudio.one.pl/) is free tool which will help you build the schema.
It also lets you export the structure and/or data as sql statements, which you can then use to build a copy of your database elsewhere.
Is this even possible? To create, say, myProgramDB.sqlite off the
command line with the SQLite tool, and then (essentially) cut-n'-paste
that file into my Eclipse/NetBeans project (of course, in the right
directory!) and have it work?
Yes of course, you can do it. Haven't you got somewhere in your code a getConnection method call? It's used to connect to the desired database. In your case should be something like:
DriverManager.getConnection("jdbc:sqlite:" + databaseName);
I just want to create the DB somewhere else, then copy it into my
project, instead of developing the DB through my app directly.
That's reasonable. The only thing that you might consider is this: if your application depends on the database "skeleton" as you said, then a database file (talking about SQLite) must always be available in order to proper run your program. Embedding inside your application, the basic instructions to create the database tables required, could permit to the application to rebuild a minimal database if the file is accidentally lost.
There are a number of GUI schema-creation and browsing clients for SQLite.
CAVEAT:
There are some differences in the way various implementations of SQLite differentiate (or don't differentiate) between INTEGER datatype and the other ways of expressing integer, such as INT, INT32, BIGINT, etc., especially when then column is a primary key.
If creating a SQLite schema outside of the implementation where you plan to use it, use "INTEGER" (verbatim) when assigning integer data type affinity to a column; do not use any of the other variants of int.
I have my most of my apps "dynamic" data stored in the datastore.
However, I also have a large collection of static data that would only change with new builds of the app. A series of flat files seems like it might be simpler than managing it in the datastore.
Are there standard solutions to this? How about libraries to make loading/parsing this content quick and easy? Does it make more sense to push this data to the datastore? Which would perform better?
Anyone else have this problem and have war stories they can share?
Everything depends on how you need to use the information.
I for instance have an application that needs to have a starting state provided from static data. Since I wanted this static data to be easily prepared outside the application, I put the data as spreadsheets on Google Docs and then I have an administrative function in my web app to load the starting state through Google Docs Spreadsheet API to objects in the datastore. It works fairly well, although there are some reliability issues that I haven't quite worked out yet (I sometimes need to restart the process).
In other cases, you might get away with just including the data as static property/xml files and load them through the standard Java resource APIs (getResourceAsStream and such). I haven't tried this approach though since it wasn't meaningful in my particular situation.
If I have systems that are based on realtime data, how can I ensure that all the information that is current is redundantly stored in a file? So that when the program starts again, it uses this information to initialize itself back to where it was when it closed.
I know of xstream and HSQLDB. but wasn't sure if this was the best option for data that needs to be a literal carbon copy.
It really all depends what type of app data you're storing. If you need to recreate java objects exactly how they were (i.e. variables and state the same), you can serialize the objects you need. There are many serialization mechanisms, for example, xstream as you mentioned. If you're storing objects directly, using one of those mechanisms would work.
But, a lot of times, you want to store the state of your application, which doesn't necessarily correspond directly to serializing objects directly. If that's the case, you can write out only the relevant data you need. The type of storage you use depends on your needs. If you have a large amount of data, consider a database. A smaller amount might work better in a flat file.
One other thing is that storing data redundantly in a single file doesn't seem too useful. If the file gets corrupted, you'll lose both copies, so if redundancy is a concern, store it in different places (i.e. a primary and backup database).
There's no one right way to do it, but hopefully these ideas get you started.
Creating a literal copy (i.e. a snapshot) of a large body of in-memory data is expensive. Repeating the process each time you get an update to the in-memory data is probably prohibitively expensive. You need to re-think your application architecture.
One approach is to commit your realtime data to a database as it comes in, and then display the data either from the database for coherency.
A second approach is to commit to a database and maintain a parallel in-memory data structure which you display from. You also need to implement code to rebuild the in-memory data structure from the database on application restart. This is more code, and there is more opportunity for glitches where the user sees different stuff after a restart due to some bug.
A third approach is to work entirely from an in-memory data structure and deal with data persistence as follows:
periodically, you suspend processing updates and take a snapshot of the entire in-memory data structure using xstream, java serialization or whatever.
every update needs to be reliably logged (with a timestamp) to a file or files in a form that can be replayed.
when the application restarts, you reload from the last snapshot and then replay all updates that arrived since the snapshot.
The last approach has the problem that there is only one up-to-date stable copy of the data. If that is lost due to a hard disc or OS failure, then you are toast. In the other approaches, this issue can be address using a hot standby database implemented using the RDBMS's off-the-shelf support for such things.
I have a simple task that I feel there has to be an app out there for (or is easy to build or extend an open-source version).
I need to run a mysql query repeatedly and look for changes in the results between runs (the data is coming in in real time).
I have built several of these queries and throughout the day find myself jumping between tabs in my mysql client running them, and trying to see what has changed. This becomes difficult as there are hundreds of rows of data and you can't remember the previous values easily.
Ideally I could have a simple app (or web app) that stores the query, and refreshes over and over again. As the data is filled into the table it could compare the old results and change the color to red or green (or something).
I would need sorting, and simple filtering (possibly with string replaces into the query based on the inputs.
We run Ubuntu at work and I have tried doing this via terminal scripts (we use Ruby), but I feel a more-visual output would give me better results.
Googling around I see several for-pay apps, but there has to be something out there to do this.
I don't mind coding one up, but I don't like to re-invent the wheel if I don't have to.
Many thanks!
For simple things like this you are not reinventing the wheel as much as making your own sandwich -- some things don't make much sense to buy. Just build the simplest web page possible (e.g. a table with the table names you are interested in and maybe a timestamp for the last time it was checked. Have some javascipt run your query and color the cells based on the change you are looking for...repeating this operation as needed. I could give you more specific info if you can tell me how the data changes...more entries into a table? Updates to existing data?
I often use JDBC servlets via Tomcat for this. Here's an excellent tutorial and a very simple example.
I've done something similar in the past using Excel. Just build a connected spreadsheet, make your queries and the result will be outputed to Excel, then you format the way you like it. Very flexible, and if you need some kind of logic beyond the query itself, there are always Excel's built in functions and VBA.
Here is a useful link to help you. It is very simple:
http://port25.technet.com/archive/2007/04/10/connecting-office-applications-to-mysql-and-postgresql-via-odbc.aspx