I have created a java daemon program that collects data from social network accounts. I use a lot of services including Flick, S3, GeoCoding, etc. Currently I have the program set up to read all these API keys from a properties file. I also have a similarly formatted properties file in my test folder that contains different keys for testing purposes. These property files are not committed to source obviously. This collection program writes to a mongo db. I am also building a web app that also works with mongo and will be deployed along side the collection. During my development I am reading that it is best to store keys as environment variables on the production side. It got me think; which leads me to my question...
I am wondering if there is a better way to handle these keys in my java program (from a deployment standpoint) or some possible routes that people have tried in doing something similar to this. Can someone shed some light on this?
I would recommend a database. If you are only storing API keys for personal use, then the size of the database isn't probably a major concern. Personally, I would suggest MySQL (or alternatively SQLite) as they are both quite well-supported.
If you encrypt your keys then it shouldn't matter too much where you store your database, although of course I still wouldn't make it openly downloadable. Just pick a good encryption tool and do not try developing your own encryption algorithm!
The latest hotness (in a world of containers) is to use zookeeper, etcd or consul as a distributed configuration store. The confd tool is capable of ensuring that application configuration files are kept in sync with changes to configuration.
My personal preference is Consul which has a similar template tool called consul-template, and another called envconsul if you would prefer your program to consume environment variables.
Finally Hasicorp, the makers of consul, have an encryption product called vault. It works well with consul and is also supported by consul-template.
Related
I'm currently getting into Socket Programming and building a multi-threaded console application where I need to register/login users. The data needs to be saved locally, but I can not seem find the right structure for it.
Here are the ideas I though about:
Simply saving the data to .txt file. (will be troublesome to search and authenticate the logins)
Using the Java Preferences API but since the application is multi-threaded I keep on overwriting the data each time a new client connects to my server. Can I create a new node for each new user?
What do you guys think is the ideal structure for saving login credentials? (security isn't currently a concern for this application)
I would consider the H2 database engine.
quote:"Very fast, open source, JDBC API Embedded and server modes; in-memory
databases Browser based Console application Small footprint: around 2
MB jar file size"
http://www.h2database.com
It really depends on what you want to do with the application. The result would be different, depending on what you would answer to the following questions:
Do you want/need to persist the databases?
Is there any other data which you need to store along with that?
are you using plain java or a framework like Spring?
Some options:
if you're just prototyping and you don't have any persistence: consider using an in-memory storage for it. For simplicity in coding/dependencies, something like a ConcurrentMap can be completely sufficient. If you wrap it properly, you can exchange it later - and you don't add dependencies and complexities at an early state.
If you're prototyping but you still need persistence, using properties files on top of the ConcurrentMaps can give you a quick win.
There might be some more stages to this, depending on where you want to go with this, choosing a database at one point can be an option. Depending on your experience and needs, you can use a SQL or NoSQL database. Personally, I get faster results with NoSQL (MongoDB in my case) but prefer SQL in production for use cases like account management.
I have an embedded system using a python interface. Currently the system is using a (system-local) XML-file to persist data in case the system gets turned off. But normally the system is running the entire time. When the system starts, the XML-file is read in and information is stored in python-objects. The information then is used for processing. My aim is to edit this information remotely (over TCP/IP) even during process. I would like to use JAVA to get this done, and i have been thinking about something to share the objects. The problem is, that I'm missing some keywords to find the right technologies to get this done. What i found is SOAP, but i think it is not the right thing for this case, is that true? I'm grateful for any tips.
As I understand, you are using XML file to store start up configuration
And my assumptions on your interface between Java & Python apps
You want your Java application to retrieve objects over Python interface
And process them locally and send it back to Python interface to reload config ?
So, depending on your circumstances, you can workout something with the following
Jython
Pickle (if you have no restriction on startup config file format or can afford to do conversion)
https://pypi.python.org/pypi/Pyro4
Also you can get some ideas from here:
Sharing a complex object between Python processes?
You should ask your python application to open a XML-RPC socket which clients can connect on. This could let an outside application to execute an endpoint, which would manipulate your python object values in someway. There are several good choices for Java XML-RPC libraries, including the amazing org.apache.xmlrpc library.
From what little understanding of Cassandra I have, it seems that data locality is mostly transparent to the client application that accesses a node, as it should.
However, what if I explicitly only wanted to access the data of a column family that is local to the node I'm connected to? Is such a thing possible? I haven't found a way of getting this from a client API out-of-the-box, but it seems that I could get some of this information through the system tables, but I can't quite figure out how to do this.
The idea is to perform mapreduce, but without using Hadoop. A local client would connect to its local cassandra node, perform aggregation on the local data and then pass it back upstream.
Is such a thing possible at all? By the looks of it, it seems possible since I've seen evidence of Hadoop being able to use Cassandra, but the examples seem to be geared towards Hadoop rather than a generic client. The local client (the bit talking to Casandra) would be in Java. I'm currently using Hector, but I'm unsure whether it would provide any data locality information.
A recent article on the Netflix Techblog introduces Aegisthus, a project which reads the SSTables stored on disk across the cluster and merges them into a single, consistent view of the data (in MapReduce). I would imagine that the mechanics would then trivially exist for generating a view of the data on a single node.
Unfortunately, I don't think they've open sourced this tool yet so you won't be able to use it. The most it can be at this point is a glimmer that yes it's possible to natively read SSTables using non-Cassandra code.
You may be able to hack something together using the Cassandra source that reads SSTables and have that feed the local client you're hoping to build. A great starting point would be looking at the source of org.apache.cassandra.tools.SSTableExport which is used in the sstable2json tool.
I have my most of my apps "dynamic" data stored in the datastore.
However, I also have a large collection of static data that would only change with new builds of the app. A series of flat files seems like it might be simpler than managing it in the datastore.
Are there standard solutions to this? How about libraries to make loading/parsing this content quick and easy? Does it make more sense to push this data to the datastore? Which would perform better?
Anyone else have this problem and have war stories they can share?
Everything depends on how you need to use the information.
I for instance have an application that needs to have a starting state provided from static data. Since I wanted this static data to be easily prepared outside the application, I put the data as spreadsheets on Google Docs and then I have an administrative function in my web app to load the starting state through Google Docs Spreadsheet API to objects in the datastore. It works fairly well, although there are some reliability issues that I haven't quite worked out yet (I sometimes need to restart the process).
In other cases, you might get away with just including the data as static property/xml files and load them through the standard Java resource APIs (getResourceAsStream and such). I haven't tried this approach though since it wasn't meaningful in my particular situation.
I am making a desktop application in java and using MSAccess in data base.
I want that if i run the setup of the application the database should be created on client machine because there can be different client using the application and how can i create the setup?
is there any tools available for this free of cost?
please explain me in detail..
thanks
Java 6 (enhanced for desktop application work) comes with a built-in database called JavaDb (formerly IBM's Derby). That will do what you want.
Here's a set of guides and tutorials on how to use it.
I would suggest that when your application first starts, it checks for the presence of the created database, and if it doesn't exist, it builds the database (via the appropriate SQL). I've used this approach before and it works quite well.
I prefer nullsoft. Take a look at Open Source Installers Generators in Java
#pratap: database should be created on client machine..
Add an empty access database to your setup.
Have a look at SQLite, which is used by Mozilla (Firefox stores all bookmarks and history in a database) and several other major applications.
When you say
access in database
do you mean Microsoft Access or access the data in a database.
I would advise against MS Access if that is the case. If not, you could either use the JavaDB or HSQLDB and the use SQL scripts to create the database. As a summary
Package the application in one of the installers (InnoSetup or NSIS are good ones)
When installing, extract all the files in proper folders
Execute the SQL scripts before first running the application to ensure the database is setup, you can do other housekeeping tasks along with this step (refer to installer documentation for after-install steps)
Your application is good to go
In the last distribution of NetBeans I used, there was a wizard to create such application. The application used the Java Persistence API to store the Data.
My option is HSQLDB since it's fast, reliable and easy to use.
In the documentation it's explained how to use the standalone database mode, this is primarily used for unit testing but it fits your case too. The good thing with this is that you just connect to the file based database without any special set up and if the files doesn't exist, they're created.
I would second the posters who recommend JavaDB.
It is absurdly easy to adminster from inside your application. Whats more because everything is native Java you dont get the char->unicode little-endian->big-endien and all the other conversion malarky you normally get when reading SQL into java.
The one tip is that with JavaDB is prepare your SQL statements. Prepared statements get cached and the resulting access program (similar to an access plan but actually a jvm program) is reused, the programs for "executed' statements are not cached.
If you are really set on MSAccess then I would suggest you package an "default.mdb" file with all your required tables defined and your classifcation tables populated. If the user's table does not exist then simply copy over the default .mdb file and open that.
I recommend the H2 database because it is simple, fast, pure Java, and small. See this page for how H2 compares to other Java databases, including those mentioned here in other answers. It has many features Derby/JavaDB and HSQLDB do not.