ok so this is a general question ... in my app on GAE I need to keep a 2 types of static data, parameters that could change like user names and passwords (for external services not my user database) so the best way is not to hard code them and keep them in the datastore so I can change them from the admin system settings panel. However, how about stuff that will NEVER change, like a list of countries for selectors in forms. Here are my options:
Option 1:
Keep them in the datastore under a dataUitls entity. However, this will have a toll on my datastore quotations.
Option 2:
Hard coded in a class. This will not affect my datastore quotations, and will serve much faster in a JSP loop as I dont have to wait for datastore transactions, however, this will substantially affect my memory and instances. Example is as follows
package system.Tools;
import java.util.ArrayList;
import java.util.List;
public class SystemConstants
{
public static String[] tmp = {"String1","String2"};
public static List<String>Countries = get(tmp);
private static List<String> get(String[]countries)
{
List<String>result = new ArrayList<String>();
for(String tmp:countries)
{
result.add(tmp);
}
return result;
}
// getters and setters for whatever other parameters I have here
}
Both will work no doubt, but I need a professional opinion (preferably someone who has tried both) telling me which is a better practice for this particular situation.
Thanks alot guys !! Keep em coming !!
Hard coded constants have far less processor overhead than Datastore queries and also use less memory. The memory overhead of Datastore access classes and objects will exceed that of a fairly large number of hard coded constants.
The only possible advantage that Datastore may have, in an extreme case, is slightly less memory usage when you need only a small subset of lots of data. A list of countries is not that extreme.
The best choice will depend on specific details of your situation. Rather than speculate, measure and compare.
It depends on your requirement. If you believe your data will NEVER change, then no barrier to hardcode. Whenever you want to add multi language support, you may end up with lots of changes due to hardcoding.
Don't be too concerned about memeory usage for just a static variable. its negligible compared to other memory usages within you program.
Also there is a third way that you can use.
write these into a .property file and bundle it with your program. at the startup u read the file and load values.
In Summary, Don't think alot about memory. If your list is in the DB, you have to load it into the memory before you use. Database solution is good if those are changeable via a GUI. property file based approach is efficient and easy to program.
You have some options, depending of the case and the data:
Harcode: if the data is not changing at all and the structure is easy and it is not growing (more and more and...), it could be ok.
Properties file: if your data change from one environment to another, it could give you advantage to get the data in the a file. Change the file is easy from one environment to another, more than recompile.
Json or similar: if the data structure could change (new properties) or it is "complex", to have all in a json is more easy to manage. Parse using Jackson or similar it is one or to lines of code if you have the DTOs/classes. And the maintenance is better than hardcode data.
Database: if the data is not changing is "static" to put in the database doesn't give you any advantage, IMO.
Of course, whatever you decide, you can parse/read in the start up of the server and write the data in the memory or memcached.
Related
There can be potentially up to 1000 strings in total. Should these be hardcoded or stored in database? These are frequently accessed because everytime user wants to register or checkout an item, they are going to need to see list of area/suburb/province/countries.
If i have bunch of Enums, i think the performance should be fast because there is a max number of strings ~1-2k max.
On the other hand, if i store them in database, there's going to be latency accessing the database as well as cpu/memory consumption.
Which option do you choose?
1000 isn't a huge amount, and I would put this information into a text file and read them into the program on start-up.
Regardless, this is data, not code, and so should not be an enum (code). Why not enum? It's a lot easier and more flexible to update/change data than it is to change code, should this need to be changed in the future.
If you will definitely be updating and changing this information with time, especially if through multiple sources, then a database is surely the way to go.
It all depends on you. There is no proper convention. Below are 3 ways along with their pros and cons.
Create a class with static final string variables.
Pros:
a. Very easy to use.
b. Developers can do look ups from within IDEs.
Cons:
a. Every time you need to add/delete something, code will have to be recompiled. However, this will not be much problem if you have ci-cd in place.
Add everything in properties file and load at runtime.
Pros:
a. Modifying things will be a breeze. No code recompilation required.
Cons:
a. This would still need re-deployment and server restart.
b. Developers will be unhappy as they will have to refer the txt file every now and then. Also this could lead to mistake if developers use wrong codes which are not present in properties file.
Use database
Pros:
a. Highly configurable.
b. No need of re-deployment.
Cons:
a. Service restart will be required.
As you can see, service restart will be required for all of them as you will definitely going to use caching in case 2 and 3. My suggestion would be to use first option if they are literally never going to change as it is quite developer friendly.
i am trying to figure out if there is a 'simple' way to store persistently a large object instance in the JVM memory to be shared and re-used for multiple runs by other programs.
I am working on netbeans using java 8. The data is some ~500 MB of serialized objects. They fit easily in the RAM but take few minutes to de-serialize from disk each time.
Currently the program load a serialized object from the local disk into memory for each run. As the data is only read from during the test, it would be optimal to hold it in memory and access it directly at each run.
We've looked into RMI but the overhead, the marshalling process and the transmission will kill the performance.
I was wondering if there is a more direct way to access data from a program running on the same JVM, like sharing memory.
The multiple runs are to test different processing / parameters on the same input data.
I am open to suggestion on the best practice to achieve this 'pre-loading', any hints would be very appreciated.
Thanks
Java serialization is never going to play well as a persistence mechanism - changes to the classes can easily be incompatible with the previously stored objects meaning they can no longer be de-serialized (and in general all object models evolve one way or another).
While suggestions are is really off-topic on SO, I would advise looking at using a distributed cache such as Hazelcast or Coherence.
While you'll still have to load the objects, both Hazelcast or Coherence provide a scalable way to store objects that can be accessed from other JVMs and provide various ways to handle long-term persistence and evolving classes.
However, neither works well with big object graphs, so you should look at breaking the model apart into key/value pairs.
An example might be an order system where the key might be a composite like this:
public class OrderItemKey
{
private OrderKey orderKey;
private int itemIdex;
...
}
And the value like this:
public class OrderItem
{
private ProductKey productKey;
private int quantity;
...
}
Where OrderItems could be in one cache, while Products would be in another.
Once you've got a model that plays well with a distributed cache you need to look at co-locating related objects (so they're stored in the same JVM) and replicating reference objects.
When you're happy with the model, look at moving processing into the cache nodes where the objects reside rather than pulling them out to perform operation on them. This reduces the network load giving considerable performance gains.
If I understood well you need to read a huge amount of data from disk and use this data only for test purpose.
So every time you run the tests you need to reload them and it slow down your tests.
If this is the situation you can also try to create a disk on memory (ram disk). So your file is saved on a disk with the performances of the ram.
Here is a link for the command ramfs to create it on linux systems
I am looking at changing the way some large objects which maintain the data for a large website are reloaded, they contain data relating to catalogue structure, products etc and get reloaded daily.
After changing how they are reloaded I need to be able to see whether there is any difference in the resulting data so the intention is to reload both and compare the content.
There may be some issues(ie. lists used when ordering is not imporatant) that make the comparison harder so I would need to be able to alter the structure before comparison. I have tried to serialise to json using gson but I run out of memory. I'm thinking of trying other serialisation methods or writing my own simple one.
I imagine this is something that other people will have wanted to do when changing critical things like this but I haven't managed to find anythign about it.
In this special case (separate VMs) I suggest adding something like a dump method to each class which writes the relevant content into a file (human readable text). This method calls dump on each aggregated object as well.
In the end you have to files from each VM, and then you can compare them using an MD5 checksum for example.
This is probably a lot of work, but if you encounter any differences, you can use diff on both files, and this will be a great help.
You can start with a simple version, and refine it step-by-step by adding more output.
Adding (complete) serialization later to a class is cumbersome. There might be tools which simplify this (using reflection etc.), but in my experience you have to tweak your classes: Exclude fields which are not relevant, define a sort order for lists, cyclic relations etc.
Actually I use a similar approach for the same reasons (to check whether a new version still returns the same result): The application contains multiple services (for each version), the results are always data transfer objects, serialization is added immediately to the DTOs, and DTOs must provide a comparison method dedicated for this purpose.
Looking at the complications and memory issues, also as you have mentioned you dont want to maintain versions, i would look to use database for comparison.
It will need some effort in terms of mapping your data in jvm to db table but once you have done that, it will be staright forward. You can dump data from one large object in db tables and then you can simply run a check from 2nd object in db.
Creating a stored proc can simplify things. This solution can support data check from any number of jvms.
I am working on an hospital application where i need to show blood group in every second page in drop down and getting values from them to the db tables
since blood group are not frequent changing entity so we are planning to create a map with key value pair and make available this map throughout application in order to avid creation of same map multiple time
my question is what can be the best way to achieve this.some of the quick options coming to my mind are
Create a map at application start up and place it in application context
Create a utility class which read a property file and fill map with these values or simply create map with exisitng blood type.
but i am not sure how effective these options are as site will have to handle a good amount to user hits in near future.
Thanks in advance
Create a utility class that loads these values on system startup or load values when the class loads. Creating a class this way would give few advantages:
You can test this class and its functionality by writing test cases for this class. (Check if things are loading properly etc)
This makes you less dependent on the context and how the context works. Makes your application less troublesome to move if you, for some reason, need to change application server.
Code becomes more readable (BloodGroupUtils.getAll() compared to Application.getContext().get("bloodGroups"); or something similar.)
On Performance, this may be a bit faster. Not necessarily though ( we would need to check several other system usage/parameters to come to that conclusion.)
Retrieving the values from a static class or the application context will have essentially the same performance. The static class might be trivially faster as you don't have to get the map out of the application context, but I can't imagine it'd be worth worrying about.
Basically what I need to know is this:
I have to show a drop down list of countries to my users each country also has a code associated to it. I will have to work with both the country and the code What would be the best approach:
-We (the dev.) are thinking about a table in our app database with this data, or XML file.
-Our "architect" says that is old school and that we should use constants in our app with a map that associates the country with the code
Please Help me feel smart
I agree with you that you should not hard code this or use constants. There are a few good options depending on yours needs:
Java Properties Files - If you just have a few key-value pairs to store, these are the simplest way and easy to use.
XML Storage - If you are looking for persistence and are looking at XML for storage, I would recommend looking at JAXB. It is part of Java 6 and will make your life easier than trying to use the DOM.
Database Persistence - If you have more data that is changing often, you could also look at storing it in a database. JPA is a great standard library for doing this. That is probably overkill for what you are looking for though.
Bottom line is hard coding is a thing of the past. There are lots of great ways to get data in quickly and easily without resorting to hard coding everything.
Countries rarely change, so adding them statically as code or a config file seems reasonable. If you don't use a database for anything else, don't add one just for this feature.
If you already have XML parsing in your app, use an XML file to define the data. It already solves all kinds of issues (for example if you need to add a second attribute per country or something).
If you don't use XML for anything else, I suggest to give it a try. It doesn't add much to your app. Otherwise, use a plain text file, maybe a CSV one.
The different methods have different advantages and drawbacks:
Database:
allows you to use the country data in queries
data can be changed without redeploying the app
editing the data requires you to write some sort of frontend or do it manually via some generic SQL browser
requires database access code, and some sort of caching strategy
Any country-based logic in the code can break when the DB changes, or has to be reflected in the DB
XML:
Very easy to edit
can be changed without recompiling the app, but changes have to be deployed somehow
Requires parsing code and some sort of caching strategy
Any country-based logic in the code can break when the XML changes, or has to be reflected in the XML
Code:
Easy to edit - for developers
Changes require compilation and deployment
Requires no extra technical layers
Code and country data can't get out of synch
All in all, the "code as data" solution is indeed the nicest, if the compile&deploy step for each change is acceptable to you. The other solutions create overhead and duplication of structure (or even logic) - and no, they don't magically make it "safe" to do last-minute changes "because it's not code". Code is data, and data is code.
In short your architect is wrong (Or at least he is if your paraphrase of his position is accurate). It shouldn't be in the code.
This data is not static; a country's name changes, a new one is founded, or some cease to exist.
As far as what mechanism, it doesn't necessarily matter. Just make sure you can retrieve the data easily, that you have unit tests, and that there is straightforward mechanism to update the data.
I think that "table solution" has more flexible approach:
1. You can manage data and connecting properties
2. You can work with table directly
3. You can create associated map, based on db table))
I would certainly not use them as constants in the code.
Names can change, while countries can be created, merge, disappear, etc.
If you are already using a database, adding this may make sense. For one, it ensures that the codes that may be stored with client data are valid in terms of your country code list. So if a country disappears but a client record still refers to it, the data stays valid.
Make sure that your UI loads and caches the list; no point making a query every time if you can avoid it.
By the way, correctly handling countries in an internationalized application is much more complicated than just dealing with renames. For example, if a country or part of a country declares independence, some countries will recognize it, while others do not.