We have a system (Java web application) that's been in active development / maintenance for a long time now (something like ten years).
What we're looking at doing is implementing a RESTful API to the web app. This web application, using Jersey, will be a separate project with the intent that it should be able to run alongside the main application or deployed in the cloud.
Because of the nature and age of our application, we've had to implement a (somewhat) comprehensive caching layer on top of the database (postgres) to help keep load down. Anyway, for the RESTful API, the idea is that GET requests will go to the cache first instead of the database to keep load of the database.
The cache will be populated in a way to help ensure that most things registered API users will need should be in there.
If there is a cache miss, the needed data should be retrieved from the database (also being entered into the cache in the process).
Obviously, this should remain transparent from the RESTful endpoint methods in my code. We've come up with the idea of creating a 'Broker' to handle communications with the DB and the cache. The REST layer will simply pass across ids (if looking to retrieve) or populated Java objects (if looking to insert / update) and the broker will take care of retrieving / updating / invalidating, etc.
There is also the issue of extensibility. To begin with, the API will be living alongside the rest of servers so access to the database won't be an issue however if we deploy to the cloud, we're going to need a different Broker implementation that will communicate with the system (namely the database) in a different manner (potentially through the use of an internal API).
I already have a rough idea on how I can implement this but it struck me that is probably a problem for which a suitable pattern could exist. If I could follow an established pattern as opposed to coming up with my own solution, that'll probably be a better choice. Any ideas?
Ehcache has an implementation of just such a cache that it calls a SelfPopulatingCache.
Requests are made to the cache, not to the database. Then if there is a cache miss Ehcache will call the database (or whatever external data source you have) on your behalf.
You just need to implement a CacheEntryFactory which has a single method:
Object createEntry(Object key) throws Exception;
So as the name suggests, Ehcache implements this concept with a pretty standard factory pattern...
There's no pattern. Just hide the initial DB services behind interfaces, build tests around their intended behavior, then switch in an implementation that uses the caching layer. I guess dependency injection would be the best thing to help you do that?
Sounds like decorator pattern will suit your need: http://en.wikipedia.org/wiki/Decorator_pattern.
You can create an DAO interface for data access, something like:
Value get(long id);
And firstly create a direct DB implementation, then create a Cache implementation which calls underlying DAO instance, in this case it should be the DB implementation.
The Cache implementation will try to get value from its own managed Cache, and from underlaying DAO if it fails.
So both of your old application or the REST will only see DAO interface, without knowing any implemntation details, and in future you can change the implementation freely.
The best design pattern for transparently caching HTTP requests is to use an HTTP cache.
Related
I have read some articles about microservices architecture, but no one takes the topic of transaction. All that they says that this is hard to do it. Maybe someone can describe how to handle this?
But not from domain side, but from technology side. Lets say that we have business case where we need to invoke two different services and both of them make some changes on database. But how to rollback if some error occurs on the second one?
Who knows some libraries or design patter for this problem?
I may not be the ultimate expert in this, but I'm sure you're heading towards the Distributed Transactions. In order to have them running, all the application service components need a common shared transaction id, and you have to make sure that every component is informed about the state of the transaction. It is asynchronous, so you'll require substantial prog skills.
Here are distributed transactions mentioned or discussed:
https://en.wikipedia.org/wiki/Distributed_transaction
http://contino.co.uk/microservices-not-a-free-lunch/
http://martinfowler.com/articles/microservices.html
It would seem people try to avoid it as it is difficult. Maybe that's why you don't find much about.
Hope this helps a step forward :-)
The best design is having isolated services: each service just do its work within its own transaction and your workflow expects failures on the single service.
If you really need to commit only if all the services are called without errors you should create an higher level service that perform those calls inside an external transaction.
The first raw thing which came to my mind after reading this question is to create every add api with a delete api with ,lets say, an extra boolean flag delFlag.
boolean flag delFlag;
For POST, it will be 0. For DELETE, it will be 1.
Now you maintain a Transaction Manager which is a super service to all your micro services.
In this service maintain the calling queue of all the services and the APIs. As and when a service fails, get the calling api and call the delete method of that service and undone whatever u have done.
PS- Just a raw thought. Correct me if you think it is wrong.
Building on top of previous answers, distributed transactions are the solution. In my opinion you don't want to build your own mechanisms for tracking global transactional state, rather you want to use some kind of product - there are several out there. I have written a lengthy blog article about solving this issue with a Java application server:
http://blog.maxant.co.uk/pebble/2015/08/04/1438716480000.html
two-phase commit can be option.Coordinator send commit request message to cohorts.Cohorts send back ok.After then coordinator sends commit message to cohorts.If any failure happpens coordinator sends rollback messages to cohorts.
You can use a workflow engine(like JBPM,Activiti) to orchestrate the logic and handle transaction failures or compensatory transactions in it to achieve data integrity. This is the similar case you use in SOA architecture with ESB,BPMN and Web services
My application sends http requests to a webservice, but because the Terms of Service limit it to one query per second it is very important for me not to send more queries than I need. I put the results of some queries into a database that I check before trying the query again but some queries results are not well suited to putting in database so I would like some sort of dumb cache that would intercept my webservice calls and if the call was a duplicate just send the results of the previous call. I would expect to be able to configure the size of the cache and have it automatically remove the oldest entry if it fills up, it would be great if the cache could be configured as a file rather than use heap memory because my application is already quite memory intensive
For a simple caching solution try Google Guava libraries. The CacheBuilder/CacheLoader could be configured to your requirements. Guava provides a simple caching solution that is more sofisticated than java's own HashMap but light weight when compared to Ehcache and others. This cache could be used in a web service request interceptor that helps to decide whether to initiate a web service call.
A good tutorial with an example for Guava cache could be found here
I am working on a desktop Java application that is supposed to connect to an Oracle database via a proxy which can be a Servlet or an EJB or something else that you can suggest.
My question is that what architecture should be used?
Simple Servlets as proxy between client and database, that connects to the database and sends results back to the client.
An enterprise application with EJBs and remote interfaces to access the database
Any other options that I haven't thought of.
Thanks
Depending on how scalable you want the solution to be, you can make a choice.
EJB (3) can make a good choice but then you need a full blown app server.
You can connect directly using jdbc but that will expose url of db (expose as in every client desktop app will make a connection to the DB. you can not pool, and lose lot of flexibilities). I would not recommend going this path unless your app is really a simple one.
You can create a servlet to act as proxy but its tedious and not as scalable. You will have to write lot of code at both ends
What i would recommend is creating a REST based service that performs desired operations on the DB and consume this in your desktop app.
Start off simple. I would begin with a simple servlet/JDBC-based solution and get the system working end-to-end. From that point, consider:
do you want to make use of conenction pooling (most likely). Consider C3P0 / Apache DBCP
do you want to embrace a framework like Spring ? You can migrate to this gradually, and start with using the servlet MVC capabilities, IoC etc. and use more complex solutions as you require
Do you want to use an ORM ? Do you have complex object graphs that you're persisting/querying, and will an ORM simplify your development ?
If you do decide to take this approach, make sure your architecture is well-layered, so you can swap out (say) raw JDBC in favour of an ORM, and that your development is test-driven, such that you have sufficient test cases to confirm that your solution works whilst you're performing the above migrations.
Note that you may never finalise on a solution. As your requirements change, and your application scales, you'll likely want to swap in/out the technology most suitable for your current requirements. Consequently the architecture of your app is more important than the particular toolset that you choose.
Direct usage of JDBC through some ORM (Hibernate for example) ?
If you're developing a stand-alone application, better keep it simple. In order to use ORM or other frameworks you don't need a J2EE App Server (and all the complexity it takes with it).
If you need to exchange huge amounts of data between the DB and the application, just forget about EJBs, Servlets and Web Services, and just go with Hibernate (or directly with plain old JDBC).
A REST based Web Services solution may be good, as long as you don't have complex data, and high numbers (try to profile how long does it takes to actually unmarshal SOAP messages back and to java objects).
I have had a great deal of success with using Spring-remoting and a servlet based approach. This is a great setup for development as well, since you can easily test your code without deploying to an web container.
You start by defining a service interface to retrieve/store your data (POJO's).
Create the implementation, which can use ORM, straight JDBC or some pooling library (container provided or 3rd party). This is irrelevant to the remote deployment.
Develop your application which uses this service directly (no deployment to a server).
When you are satisfied with everything, wrap your implementation in a war and deploy with the Spring DispatcherServlet. If you use maven, it can be done via the war plugin
Configure the desktop to use the service via Spring remoting.
I have found the ability to easily develop the code by running the service as part of the application to be a huge advantage over developing/debugging something running on a server. I have used this approach both with and without an EJB, although the EJB was still accessed via the servlet in our particular case. Only service to service calls used the EJB directly (also using Spring remoting).
As we are in the beginning phases of rejuvenating our application in to SOA design I have some questions that I can not get a clear answer/picture on.
I have been doing a lot of reading, mostly around books from Thomas Erl and following that design pattern of understanding what Task Services, Entity Services and Utility Services are.
What I am stumbling on is the whole DAL concept of how that would look. So this is more of a verification of understanding or a clarification so as to help make the best approach for our platform.
So background. We currently have several web based e-commerce applications that have been pretty much been built in silos and are again pretty much a copy of each other. We have supporting applications such as Daemons and misc web services out there. Many of these applications are older then 5 years and are build on only technology (Model 1). All of our applications are centered around conducting auction sales. So during a sale event we will be taking bids from users, determine who is winning and display that information back. Each sale event has a set amount of time that they will be available to the users.
The company is moving towards a SOA solution as a lot of things we end up doing can be shared across not only our group but across other groups.
So what I understand on the DAL is that it in itself is a service which will sit on top of Data, in this case different Databases - MSSQL, ORACLE, MSSQL. Each of these databases have different schema's (Oracle) etc.
So the services (Task, Entity, Utility and Presentation Tier if needed) will make calls to the DAL to retrieve data. It is the responsibility of the DAL to know, from the contents of the message to determine what it needs to do in order to fulfill the request.
So for example, we have a Security Service candidate. This service needs to authenticate with LDAP and to authorize from the data that is stored for that given application.
The thought here is that a Utility service will be created to wrap up all the operations required to communicate with LDAP and that the Security Service will call upon the Utility Service and to the DAL to fetch the authorization data. The DAL then has the responsibility to go to the correct database/schema to retrieve the information. The information will be in XML format (standard SOA communication).
So, am I on the right track here? Have others done similar things or not? What other things do I need to consider (Currently getting the statistics on how many bids we take in an hour - on average).
Should each service have its own DAL - for example should the Security Service have the DAL as part of the service or should DAL be a shared service in which all services can use?
In your case, the approach to use for a full SOA based deployment would be to use an ESB, Identity provider and a data services solution.
To break it down, the DAL should be implemented using data services, in this way, this service will be a globally accessibly service in a language neutral way, and will support re-use and loose coupling. So all your data access logic can be implemented as web service operations in a data service.
So for the authentication and authorization management, in the SOA world, there's a standard called XACML, which is used for fine grained authorization management. So what you will need is an XACML server, who would authorize the user according to a specific criteria, where this should also have the ability to authenticate with LDAP.
Then your "Security Service" will be implemented in a service at the ESB, where that service will query the identity provider for authentication/authorization and according it's response, it will call the appropriate operations in the data service, with suitable parameters to fetch the data, and return it to the user.
The above scenarios can be implemented using WSO2 Data Services Server, WSO2 Identity Server and WSO2 ESB respectively, which are open source products, and can be freely used and found here.
i once worked with (developing) an soa project that used a "data service". it was some time ago, and i was only involved marginally, but my recollection was that it ended up being too complicated and slow.
in particular we had no real need for a data service - it would have made more sense to have placed the same abstractions in a library layer, which would have given better efficiency and no real loss of functionality (for our particular needs). this was exacerbated by the fact that the data tended to be requested in many small "chunks".
i guess it comes down to the trade-offs involved in the implementation. in our case, with a relatively closed system and a single underlying database technology, we could have easily exploited the support for distributed access that the database provided; instead we ended up duplicating this in a slower, more general, message bus, which added nothing except complexity. but i can easily imagine different cases where access to data is more "distant".
How you have to use SOA for your design is depends on the its requirements.
In generally you can write coarse grain services and expose them as web services. In your case you can write some services which calls the databases and produce the results. In this case authorization logic can also be written with the service logic.
The other approach is to use an ESB or BPEL engine to write the integration logic and expose the integrated service as a web service. In this case you can use some data services to expose data base data in xml format and integrate them. You can use services for different sachems and call the correct service with the request data. And the authorization logic can also be added to service integration logic.
Security aspects such as authentication, confidentiality, integrity is considered as non functional requirements and hence can be engaged to any service without writing an explicit security service.
Following articles describes such sample possible integration of services as mentioned in the second approach.
http://wso2.org/library/articles/2011/05/integrate-business-rules-bpel
http://wso2.org/library/articles/2011/06/securing-web-service-integration
our product is built on a client-server architecture, with the server implemented in Java (we are using POJO's with Spring framework). We have two API levels on the server:
the external API, which uses REST web services - useful for external clients and integrations with other servers.
the internal API, which uses pure Java classes - useful for the actual code inside (as many times the business logic invokes an API call) and for integration with plusins developed inside out company and deployed as parts of our product. The external REST API also uses the internal API.
We implemented permission checking (using Spring security) in the internal API because we wanted to control access at the lowest API level.
But here comes the problem: there are some operations defined on the API level that are regarded as forbidden for a currently logged user, but which should be performed smoothly by the server itself. For example, deleting some entity could be forbidden for the user, but the server might want to delete this entity as a side effect of some other operation performed by the user and we want this to be allowed.
So what is the best approach for allowing the server to perform an operation (in some kind of super-user mode) that might be forbidden for the actual logged-in user?
As I see it we have several options each of which have its pros and cons:
Implement permission checking in external level API (REST) - bad because plugins will bypass permissions checks.
Turn off permission checking for the current thread after the request was granted - too dangerous, we might allow too many server actions that should be forbidden.
Explicitly ask the internal API level to perform the operation in the privileged mode (just like PrivilegedAction in java security framework) - too verbose.
As none of the above approaches is ideal, I wonder what is the best-practice approach for this problem?
Thanks.
Security is applied at the bounds of a module. If I understand you, your system applies security on two levels of abstraction of the (roughly) same API. It sounds complex, as you have to make a double security check on the whole two APIs.
Consider migrating the REST needed methods from the internal API to the external one, and deleting security stuff in the internal API.
external API will manage security for external clients (at the boundaries of your app)
internal API will be strictly reserved for internal app and plugin use (and you would happy hack it, as no external clients are bounded to it)
Do you really need to control the plugin's permissions to your application logic ? Is there a good reason for it ? Plugins are developped by your company, after all. Maybe a formal document explaining to plugin's developpers what should not be done, or a safety test suite validation for the plugin (e.g. assert plugin does not call "this" method) will do the job either.
If you still need to consider these plugins as "untrusted", add the methods they need to your external API (on your app boundary) and create specific security profile for each use: "restProfile", "clientProfile" & "pluginProfile". Each will have specific rights on your external API methods.
It sounds like you need two levels of internal API, one exposed to plugins and one not.
The best way of enabling that would be using OSGi (or Spring Modules). It allows you to explicitly state which packages and classes can be accessed by other modules (ie REST modules and plugin modules). Those would be the exposed level of your new internal API and you would use Spring Security to further restrict access selectively. The internal packages and classes would contain the methods which did all the low level stuff (like deleting entities) and you wouldn't be able to call them directly. Some of the exposed API would just duplicate the internal API with a security check, but that would be ok.
The problem with the best way is that Spring Modules strikes me as still a bit too immature even to put into a new webapp project. There's no way I'd want to shoehorn it into an old project.
You could probably achieve something similar using Spring Security and AspectJ, but it strikes me that the performance overhead would be prohibitive.
One solution that would be quite cool if you could re-architect your system would be to take tasks requiring security elevation offline, or rather make them asynchronous. Using Quartz and/or Apache Camel (or a proper ESB) you could make the "delete my account" method create an offline task that can at a future date be executed as an atomic unit of work with admin priveliges. That means you can cleanly do your security checks for the user requesting account deletion in a completely separate thread to where the deletion actually takes place. This would have the advantage of making the web thread more responsive, although you'd still want to do somethings immediately to preserve the illusion that the requested action had been completed.
If you're using Spring, you may as well utilize it fully. Spring offers AOP that allows you to use interceptors and perform these cross-system checks, and in the event of an unauthorized action, prevent the action.
You can read more about this in Spring's online documentation here.
Hope this helps...
Yuval =8-)