I have a 2-tier application (a heavy client written in C++ which connects to the object-oriented database). The database itself is InterSystems Cache', and it is actually both a database and an application server (Cache' is also a MUMPS interpreter).
For performance reasons, I want to design a client-side cache (or, more generally, a persistence manager).
InterSystems Cache' does have "fast" interfaces like ODBC/JDBC, but I'm dealing with a lot of legacy client code which has already been using Object Binding for ages. So I can't change the architecture of the client, but have to make the protocol faster. The protocol itself is originally very verbose: all class/method/property names are sent verbatim, so, for instance, creating a single object server-side "costs" me 50k traffic.
Classes at the server side support inheritance and can have properties and methods. So using Cache' Object Binding means I can:
create and delete objects,
read and update properties, and
call methods.
What is important here, a server-side method call generally executes the code, the nature of which is unknown to the client. Since this code may potentially alter the state of the objects in the database, client-side cache may need to be invalidated after a method call. This is different from regular CRUD operations, where the client can keep track of changes he made to the objects, and update the cache accordingly.
Questions:
Which Java persistence managers are worth looking at, so that I can take the idea and reinvent the wheel? I'm thinking of J2EE entity beans, JPA, and in-memory grids like Coherence.
Which C++ persistence managers can be adapted to use the InterSystems API? Particularly, are Protocol Buffers a fit for my task?
Which methods can be used to "compress" the protocol which is originally very verbose on the wire? My first call is ZIP-compressing the traffic and hashing (encoding) class/method/property names (so that a TLV structure containing integers instead of names is sent over the network). Any other ideas?
What reading on Enterprise Patterns (particularly, in C++) applicable in my case can you suggest?
ORM framework (Wt::Dbo) seems to do this job:
http://www.webtoolkit.eu/wt/doc/tutorial/dbo/tutorial.html
Wt looks like a very interesting tool to implement a modern web application in C++.
Related
I am researching how to build a general application or microservice to enable building workflow-centric applications. I have done some research about frameworks (see below), and the most promising candidates share a hard reliance upon RDBMSes to store workflow and process state combined with JPA-annotated entities. In my opinion, this damages the possibility of designing a general, data-driven workflow microservice. It seems that a truly general workflow system can be built upon NoSQL solutions like MondoDB or Cassandra by storing data objects and rules in JSON or XML. These would allow executing code to enforce types or schemas while using one or two simple Java objects to retrieve and save entities. As I see it, this could enable a single application to be deployed as a Controller for different domains' Model-View pairs without modification (admittedly given a very clever interface).
I have tried to find a workflow engine/BPM framework that supports NoSQL backends. The closest I have found is Activiti-Neo4J, which appears to be an abandoned project enabling a connector between Activity and Neo4J.
Is there a Java Work Engine/BPM framework that supports NoSQL backends and generalizes data objects without requiring specific POJO entities?
If I were to give up on my ideal, magically general solution, I would probably choose a framework like jBPM and Activi since they have great feature sets and are mature. In trying to find other candidates, I have found a veritable graveyard of abandoned projects like this one on Java-Source.net.
Yes, Temporal Workflow has pluggable persistence and runs on Cassandra as well as on SQL databases. It was tested to up to 100 Cassandra nodes and could support tens of thousands of events per second and hundreds of millions of open workflows.
It allows to model your workflow logic as plain old java classes and ensures that the code is fully fault tolerant and durable across all sorts of failures. This includes local variable and threads.
See this presentation that goes into more details about the programming model.
I think the reason why workflow engines are often based on RDBMS is not the database schema but more the combination to a transaction-safe data store.
Transactional robustness is an important factor for workflow engines, especially for long-running or nested transactions which are typical for complex workflows.
So maybe this is one reason why most engines (like activi) did not focus on a data-driven approach. (I am not talking about data replication here which is covered by NoSQL databases in most cases)
If you take a look at the Imixs-Workflow Project you will find a different approach based on Java Enterprise. This engine uses a generic data object which can consume any kind of serializable data values. The problem of the data retrieval is solved with the Lucene Search technology. Each object is translated into a virtual document with name/value pairs for each item. This makes it easy to search through the processed business data as also to query structured workflow data like the status information or the process owners. So this is one possible solution.
Apart from that, you always have the option to store your business data into a NoSQL database. This is independent from the workflow data of a running process instance as far as you link both objects together.
Going back to the aspect of transactional robustness it's a good idea to store the reference to your NoSQL data storage into the process instance, which is transaction aware. Take also a look here.
So the only problem you can run into is the fact that it's very hard to synchronize a transaction context from a EJB/JPA to an 'external' NoSQL database. For example: what will you do when your data was successful saved into your NoSQL data storage (e.g. Casnadra), but the transaction of the workflow engine fails and a role-back is triggered?
The designers of the Activiti project have also been aware of the problem you have stated, but knew it would be quite a re-write to implement such flexibility which, arguably, should have been designed into the project from the beginning. As you'll see in the link provided below, the problem has been a lack of interfaces toward which to code different implementations other than that of a relational database. With version 6 they went ahead and ripped off the bandaid and refactored the framework with a set of interfaces for which different implementations (think Neo4J, MongoDB or whatever other persistence technology you fancy) could be written and plugged in.
In the linked article below, they provide some code examples for a simple in-memory implementation of the aforementioned interfaces. Looks pretty cool and sounds to perhaps be precisely what you're looking for.
https://www.javacodegeeks.com/2015/09/pluggable-persistence-in-activiti-6.html
I'm currently stuck between two options:
1) Store the object's information in the file.xml that is returned to my application at initialization to be displayed when the GUI is loaded and then perform asynchronous calls to my backend whenever the object is edited via the GUI (saving to the file.xml in the process).
-or-
2) Make the whole thing asynchronous so that when my custom object is brought up for editing by the end-user it queries the backend for the object, returns the xml to be displayed in the GUI, and then do another asynchronous call for if something was changed.
Either way I see many cons to both of these approaches. I really only need one representation of the object (on the backend) and would not like to manage the front-end version of the object as well as the conversion of my object to an xml representation and then breaking that out into another object on the flex front-end to be used in datagrids.
Is there a better way to do this that allows me to only manage my backend java object and create the interface to it on the front-end without worrying about the asynchronous nature of it and multiple representations of the same object?
You should look at Granite Data Services: http://www.graniteds.org If you are using Hibernate: it should be your first choice, as BlazeDS is not so advanced. Granite implements a great facade in Flex to access backend java objects with custom serialization in AMF, support for lazy-loading, an entity cache on the flex-side with bean validation. Globally, it is a top-down approach with generation of AS3 classes from your java classes.
If you need real-time features you can push data changes on flex client (Gravity module) and solve conflicts on the front side or implement conflict resolvers on the backend.
Still you will eventually have to deal with advanced conflicts (with some "deprecated" flex objects to work with on the server: you don't want to deal with that), a basic feature for instance is to add a version field and reject manipulation of such objects on the backend automatically (many ways to do that): you will have to implement a custom way for a flex client to update itself to the current changes implying that some work could be dropped (data lost) on the flex client.
If not so many people work on the same objects on your flex application, this will not happen a lot, like in a distributed VCS.
Depending on your real-time needs (what is the frequency of changes of your java object? This is the most important question), you can choose to "cache" changes in the flex side then updating the whole thing once (but you'll get troublesome conflicts if changes have happened) or you can check everytime the server-side (granite enables this) with less conflicts (and if one happens: it is simpler) but you'll generate probably more code to synchronize objects and more network traffic.
I planning to split my systems into front-end and back-end. Currently my application directly communicates with database, but I want to create a Spring Web service to do it instead. My problem lies with using Hibernate to map my objects to database tables.
I need my front end program to have persistant up-to-date interaction with the database. This again means I have to write a lot of web service endpoints to handle all the queries and updates. This again makes it Hibernate mapping pointless, since I'm not gaining anything.
My question is: is there a proven and reasonable way to pass (via SOAP if possible) hibernate mapped objects over to front-end and later commit changes done to these objects?
In short: no.
Detaching and re-attaching hibernate-managed objects in different applications, like you are thinking of, will lead to all kinds of problems that you want to avoid, such as concurrency and locking issues, after you've dealt with all the LazyLoadingExceptions. It will be a pain in the b***.
The road you're heading into finally leads to an architecture that adds an extra layer of indirection with Data Objects being transferred between business service and clients of those business services. Only your business service will be able to talk to the database directly. Obviously this time-consuming, and must be avoided if possible. That's why I asked you to explain the problem you're trying to solve.
You can pass hibernated entities via SOAP or other serialization mechanisms, but you shall be very careful with lazy loading, collections loading and detaching entities from session - otherwise you may end up sending all your database where you need just one object or hibernate proxies which are not usable on the other side.
Quick question on what is the best practice for integrating with external systems.
We have a system that deals with Companies which we represent by our own objects. We also use an external system via SOAP that returns a Organization object. They are very similar but not the same (ours is a subset of theirs).
My question is, should we wrap the SOAP service via a Facade so we return only Company objects to our application, or should we return another type of object (e.g. OrgCompany), or even just use the Organization object in our code.
The SOAP service and Organization object are defined by an external company (a bank), who we have no control over.
Any advice and justification is much appreciated.
My two cents, Introducing external objects into application is always a problem. Especially during maintenance. A small service change might lead into big code change in the application.
It's always good to have a layer abstraction between the external service and application. I would suggest to create a service layer which will do the translation of external service object to your application domain objects and use them within the application. A clear separation / decoupling helps a lot in maintenance.
The below diagram depicts the above content.
Your decision here is how you want to manage external code dependencies in your application. Some factors that should play into your decision:
1) How often will the API change, and what's the expected nature of the changes?
2) What's the utility of your application outside its depdencies? If you removed the SOAP service dependency, would your app still serve a purpose?
A defensive approach is to build a facade or adapter around SOAP service, so that your code only depends on your object model. This gives you a lot of control and a relatively loose coupling between your code/logic and the service. The price that you pay for this control is that when the SOAP contract changes, you must also usually also change a layer of your code.
A different approach is to use the objects you're getting from the WSDL directly. This is beneficial when it doesn't make sense to introduce a level of indirection in your application between the client code, i.e. your application is just a feeder into a different system and the whole point of the app is to stuff the Organization object into a JMS pipeline or something similar. If the SOAP API contract never changes and you don't expect the output of your app to change much, then introducing an extra layer of indirection will just hinder the readability of your codebase long term.
Most j2ee developers tend to take the former approach in my experience, both because of the nature of their applications, and wanting to separate their application logic from the details of the data source.
hope this helps.
I can't think of any situation where it's good to use the objects that another company controls. The first thing you should do is bridge those objects into your own. Also, by having your own objects, you can expand their functionality beyond the one that is provided by the third party you connect to (for example if in the future you need to talk to more than one Company object provider)
Look at the Adapter pattern.
I'd support Sridhars suggestion, I'd like just to add that for translating external service objects to your application domain you can use Dozer :
http://dozer.sourceforge.net/documentation/mappings.html
I typically always Adapt externally defined domain objects to an internal representation.
I also create a comprehensive suite of tests against the external domain object, that will highlight any problems quickly if the external vendor produces a new release.
The Enterprise service bus Architecture might be useful here
Its primary use is in Enterprise Application Integration of
heterogeneous and complex landscapes.
(from Wikipedia)
I would check out open source Mule if you are looking for an open source solution
I'm a .NET Developer trying my hand at Java. My current project has a UI layer, Business logic layer, and a Data Access layer. I'm currently working on the DAL.
I'm not connecting to an external database yet; I had hoped to have my DAL classes utilize in-memory dataTables until the DB is in place.
In .NET it's very easy to make in-memory dataTables, select from them, add to them, and remove from them. But, in Java, I've been unable to find something that does the same thing.
I was considering replacing the 'dataTables' with a collection of strongly typed objects; but that would require adding references to Business layer inside of the DAL (and I thought that was a no-no).
Can someone help a confused developer out? If this whole approach is flawed, what would you do? If I missed the equivalent of a dataTable in Java - what is it?
Here's an article on running an in-memory Derby database.
If I knew what database and what persistence library you're using, I might be able to give a more precise answer.
You could use a memory database like described in this answer.
A comparison of different memory databases is shown in this SO question.
I was considering replacing the
'dataTables' with a collection of
strongly typed objects; but that would
require adding references to Business
layer inside of the DAL (and I thought
that was a no-no).
Who makes up these rules?
If your data access layer is responsible for CRUD operations for model objects, it seems to me that it has to have references to them. There's no way around that.
The persistence tier need not know about the service or view layers.
The only completely uncoupled class is one that talks to no one and offers nothing. It's useless.
Don't be so hung up on "rules". You're trying to layer your application. You're putting all things about persistence into a layer of classes.
I don't think in-memory database has any effect on the way you design the persistence tier. You should be able to swap in a relational database or flat file or any other mechanism, but the interface shouldn't change. That's an implementation detail.
OR/Ms were available much earlier in Java than in .NET. DataSets are flawed in that they force you to program procedurally. Try to interact with objects and map those to the DB later.