We Keep Coding

Application upgrade from monolithic to microservices

We have 13 years old monolithic java application using
Struts 2 for handling UI calls
JDBC/Spring JDBC Template for db calls
Spring DI
Tiles/JSP/Jquery for UI
Two deployables are created out of this single source code.
WAR for online application
JAR for running back-end jobs
The current UI is pretty old. Our goal is to redesign the application using microservices. We have identified modules which can run as separate microservice.
We have following questions in our mind
Which UI framework should we go for (Angular/React or a home grown one). Angular seems to be very slow and we need better performance as far as page loading is concerned.
Should UI/Javascript make call to backend web services directly or should there be a spring controller proxy in deployed WAR which kind of forwards UI calls to APIs. This will also help if a single UI calls requires getting/updating data from different microservice.
How should we cover microservice security aspect
Which load balancer should we go for if we want to have multiple instance of same microservice.
Since its a banking application, our organization does not allow using Elastic Search/Lucene for searching. So need suggestion for reporting using Oracle alone.
How should we run backend jobs?
There will also be a main payment microservice which will create payments. Since payments volume is huge hence it will require multiple instances. How will we manage user logged-in session. Should we go for in-memory distributed session store (may be memcache)

This is a very broad question. You need to get a consultant architect to understand your application in depth, because it is unlikely you will get meaningful in-depth answers here.
However as a rough guideline here are some brief answers:
Which UI framework should we go for (Angular/React or a home grown one). Angular seems to be very slow and we need better performance as far as page loading is concerned.
That depends on what the application actually needs to do. Angular is one of the leading frameworks, and is usually not slow at all. You might be doing something wrong (are you doing too many granular calls? is your backend slow?). React is also a strong contender, but seems to be losing popularity, although that is just a subjective opinion and could be wrong. Angular is a more feature complete framework, while React is more of a combination of tools. You would be just crazy if you think you can do a home grown one and bring it to the same maturity of these ready made tools.
Should UI/Javascript make call to backend web services directly or
should there be a spring controller proxy in deployed WAR which kind
of forwards UI calls to APIs. This will also help if a single UI calls
requires getting/updating data from different microservice.
A lot of larger microservice architectures often involve an API gateway. Then again it depends on your use case. You might also have an issue with CORS, so centralising calls through a proxy / API gateway, even if it is a simple reverse proxy (you don't need to develop it) might be a good idea.
How should we cover microservice security aspect.
Again no idea what your setup looks like. JWT is a common approach. I presume the authentication process itself uses some centralised LDAP / Exchange or similar process. Once you authenticate you can sign a token which you give to the client, which is then passed to the respective micro services in the HTTP authorization headers.
Which load balancer should we go for if we want to have multiple
instance of same microservice.
Depends on what you want. Are you deploying on a cloud based solution like AWS (in which case load balancing is provided by the infrastructure)? Are you going to deploy on a Kubernetes setup where load balancing and scaling is handled as part of its deployment fabric? Do you want client-side load balancing (comes part of Spring Cloud)?
Since its a banking application, our organization does not allow using
Elastic Search/Lucene for searching. So need suggestion for reporting
using Oracle alone.
Without knowledge of how the data on Oracle looks like and what the reporting requirements are, all solutions are possible.
How should we run backend jobs?
Depends on the infrastructure you choose. Everything is possible, from simple cron jobs, to cloud scheduling services, or integrated Java scheduling mechanisms like Quartz.
There will also be a main payment microservice which will create
payments. Since payments volume is huge hence it will require
multiple instances. How will we manage user logged-in session. Should
we go for in-memory distributed session store (may be memcache)
Not really. It will defeat the whole purpose of microservices. JWT tokens will be managed by the client's browser and expire automatically. You don't need to manage user logged-in session in such architectures.

As you have mentioned it's a banking site so security will be first priory. Here I have few suggestions for FE and BE.
FE : You better go with preactjs it's a react like library but much lighter and fast as compare to react. For ui you can go with styled components instead of using some heavy third party lib. This will also enhance performance and obviously CDNs for images and big files.
BE : As per your need you better go with hybrid solution node could be a good option.e.g. for sessions.
Setup an auth server and get you services validate user from there and it will be used in future for any kinda service .e.g. you will expose some kinda client API's.
User case for Auth : you can use redis for session info get user validated from auth server and add info to redis later check if user is logged in from redis this will reduce load from auth server. (I have used same strategy for a crypto exchange and went pretty well)
Load balancer : Don't have good familiarity with java but for node JS PM2 will do that for you not a big deal just one command and it will start multiple instances and will balance on it's own.
In case you have enormous traffic then you better go with some messaging service like rabbitmq this will reduce cost of servers by preventing you from scaling your servers.
BE Jobs : I have done that with node for extensive tasks and went quite well there you can use forking or spanning this will start a new instance for particular job and will be killed after completing it and you can easily generate logs along with that.
For further clarification I'm here :)

what technology to use for a database service?

I am going to make a small trade management system. I want to make a independent database service to which all the other client connect. The database will be MYSQL and I will be using Java for making the service. The client can either be a Web Application or a desktop application using Java Swing (has not decided yet). There will be another layer sit between the client and the database service to handle the business logic (I call it trade service).
The architecture is something like: Client -> Trade Service -> Database Service.
My questions is that what client/service communication technology is the best suitable one for client->Trade Service and the best suitable one for Trade service -> Database.
Shall I make it s RESTful service? SOAP? Using RPC? Or any other technologies?
Many thanks for your help. Any idea or suggestions are welcome.

Take a look at RabbitMQ, A pool messaging service
http://www.rabbitmq.com/
It's Robust, flexible, fast and scalable and you can use it to communicate in Java, PHP, or whatever technologies ou want.

Shall I make it s RESTful service? SOAP? Using RPC?
These are all very similar approaches in that they are over HTTP so - assuming that's what you want; I would recommend using RESTful. You'll have lot's of examples to work with and it will allow you flexibility in the future to do things like switching out the UI layer for a smart phone app or desktop app.
Regardless of what model you pick you should understand how it works first and build in things like security and guidelines early. Do your homework now. Trying to change the middle layer of a design like this is a pain.

There is no blanket answer to your question, there are instead options based on your skill set. Do you conceptually understand the HTTP spec completely and be able to extend it to REST, that works very closely with HTTP (common creation ancestor). Do you better understand the traditional method invocation of SOAP? Are you tied in your ecosystem to a specific language, as this can impact which tools you choose from.
If you were paying me to write a service based on the simple requirements you have given (which is nearly impossible), I would create a domain driven design service (your business layer) with a RESTful interface and Spring JDBC for data access. That is me, and what I work in most often. My partner in crime at work would probably choose SOAP and Hibernate.

I think what you're taking about is Queues, and I'm guessing you need a managed service for that. Queues can be the glue between your micro-services. Some of the vendors I know which have Queue as a Service are :
CloudBoost.io : https://www.cloudboost.io
Check out https://tutorials.cloudboost.io/en/queues/basicqueues for documentation.
Iron.io : https://www.iron.io
P.S : I work at CloudBoost

RMI vs REST Service

We are developing a service for our portal / web client developed using JSF . My advice was to expose the service as REST but another team member said to go with RMI implementation since its easier to deal in java object from development and testing point of view.
My argument was development and testing efforts with be pretty much the same but we will get all the goodness of REST web services.
FYI : We already have REST setup so there is no extra cost in framework support. This services are exposed for our smartphone client who uses REST api.
At the end our Manager decided to go with RMI way but I still think REST would be smarter way.
What would be your choice REST or RMI?
Note : Nothing against my team member or Manager just trying to learn here.

If there are firewalls between your client and server, it is likely that RMI traffic might be blocked. HTTP traffic is open on most firewalls and REST should have no problem getting through.

The biggest argument against RMI and for REST/SOAP etc is that the client does not have to be Java.
If your front-end could change down the road from JSF to ASP, then you'll be in some trouble.
Other than that, RMI is the way to go. An even better way to go is EJB ( which is a layer on top of RMI ) with additional advantages -- lots of vendors already implement the EJB spec, you get the advantages of object pooling, transaction management etc.

Is the client in Java, use RMI. But that is to simple thought. As it is only a point to point protocol.
REST as a paradigm is interesting if you have e.g. many reads and like to use HTTP technologies for caching etc. The next thing is you likely can easy implement "paging cursors" so you send data as a small page and add info how to retrieve the next page.
Your question basically is formulated as if it is a technology question. Which is the wrong approach. You should not bother about technology but about system architecture. The whole software system, its abilities, its performance, its scaling, its configuration and its maintenance is completely different depending on your usage of RMI or REST.

Advice on Application Architecture

I am about to build a system that will have its own engine, as well as a front end user interface. I would like to decouple these two as much as possible. The engine should be able to accept commands and data, be able to work on this data, and return some result. The jobs for the engine may be long, and the client should have the ability to query the engine at any time for its current status.
A decouple front-end / back-end system is new territory for me and I'm unsure of the best architecture. I want the front end to be web-based. It will send commands to the engine through forms, and will display engine output and current status, all through ajax calls. I will mot likely use a Spring-based web app inside Tomcat.
My question involves the best structure for engine component. These are the possibilities I'm considering:
Implement the engine as a set of threads and data structures within the web app. The advantages here would be a more simple implementation, and messaging between the web app front end and engine would be simple (nothing more than some shared data structures). Disadvantages would be a tight coupling between the front and back ends, reliance on the server container to manage the engine (e.g if the web server or web app crashed, so would the engine).
Implement the back end as a stand alone Java application, and expose its functionality through some service on a TCP port. I like this approach because it's decoupled from the web server. However, I'm not stoked about the amount of low-level networking / communication code required. I would prefer some higher level of message passing that abstracts Sockets etc.
Use an OSGi container like Spring DM server to host both the web app and engine. This approach is nice because the networking code is nonexistent. The engine exposes services to the OSGI container for the web app to consume. The downside here is the learning curve and overhead of a new technology: OSGi. Also, the front and back end remain coupled again which I dont really want. In other words, I couldn't deploy the front end on any old servlet container, it would have to be in the same OSGi container as the engine.
I have a feeling RMI is the way to go here, but again that's a new area of technology for me, and it still doesn't explain how to design the architecture of the underlying systems. What about JMS?
Thank you for any advice.

If you really want to decouple web app and engine,you can also deploy the engine in a different server and expose the API as web service calls (WS-* or REST).

If it's going to be a Web app, there's no need to decouple the processes like there would be if you had a desktop app front end and a server back end. So keep it simple.
The basis I would use (and am using for a project I'm working on currently as it turns out) is this kind of stack:
Spring 3
Web container
Application deployed as a Web application (WAR);
For persistence, either Ibatis (my preferred option) or JPA/Hibernate (if you prefer a more object persistence approach);
Your preferred choice of Web framework. There's no easy answer here and there are dozens to choose from, from the straight templating to the more componentized (JSF, Seam, etc). Tapestry/Wicket look interesting but I'm no expert in either.
A Spring container is entirely capable of launching a series of threads and it's quite common to do so. So what you'll need is a series of components that will simply be your engine. Unless you have a good reason to do otherwise, Spring beans within a Web application context is simple, flexible and powerful.
On the front end it depends on what you want. Straight HTML can be done with any Web framework. Even if decorated by some Javascript. I use jQuery for that kind of thing.
It only gets a little different if you want the front end to look like a desktop app (a so-called "rich" UI). For this you either need to use the Google Web Toolkit ("GWT"), possibly a component Web framework like JSF (although I tend to think these get real messy real fast) or a Javascript framework like ExtJS, SmartClient, YUI or the fairly new Uki.

You'll decouple your UI if you write your back end as services, and establish an XML or JSON message format to pass between client and services.
All the rest of cletus's comments can hold true for the back end, but the client can be blissfully unaware of it. It can even be a .NET implementation for all it cares. The focus is on the use cases and the messages, not the back end implementation.
This can also be useful in those instances when you use a non-HTML based UI (e.g., Flex).

AJAX Using a REST API Without Server Side Framework

I am building a REST based API (Read only) for accessing services of my application. I plan on writing a web application that will leverage these APIs to provide basic information on the status of the application. I plan to use AJAX (using jQuery) to show the information.
Originally I planned on using Grails, Spring MVC, RoR or one of the web frameworks to handle the back end of my application. The REST APIs I will be providing though are already built on a stanalone REST framework so I would only be leveraging the web framework for the core application and not the business logic. In the future, I might need the server side framework to handle other tasks but for now most of the work is done in the REST APIs.
My question is, should I bother with using a web application framework on the server side? I should be able to make all the API calls I need from AJAX directly from the browser. I cannot think of much I would need to do on the server side. Would it make sense to have the application be standard HTML + AJAX + REST?

Its hard to say without actually knowing more about your current setup. This is what your situation sounds like:
You already have an app ready to go with all of the business logic contained. Sounds like its written in Java?
You already have the services written and exposed through a REST api using some other standalone framework. Meaning that if you wanted to, you could access the data right now with the browser without any extra work.
You have not yet built the web application, but when you do, it will get all of its content from the REST api using XHR and jquery. I say that, because otherwise, I would think that you would already be using some kind of framework to generate the other content.
If I am correct in my assumptions, then I would say that you have no need for an additional framework layer. Grails, RoR, SpringMVC my use ajax, and aid in exposing REST services, but the bulk of what they provide is an easy way to make an application that must generate html on the server, deal with form submissions, and handle sessions in a request/response cycle. It doesn't really sound like you'll be doing any of that, and it will likely make your app more complicated.
If you did at some point need the things that rails etc. provides, I would say that you may not need to use rails to expose the rest apis you have now. You could use rails just for what you need, and continue to use what you have for the REST api.

Well, the AJAX calls need to pull data from a server somewhere. If the goal is to avoid a complicated setup on the server side, CherryPy can keep the server side code VERY small.
I've written a simple exapmle below. The first class would be where you put the logic for your ReST API. The code below the class is all you need to get the server up and running.
Install Python 2.6, save the code below to restExample.py. Then in your command line run the python file by doing "python restExample.py". Point your browser to http://localhost:8080/blog/999 and see some JSON come back.
import cherrypy
import json
# Create the controller
class Blog_Controller(object):
def get(self, entryID):
cherrypy.response.headers['Content-Type'] = 'application/json'
return json.dumps({'title':'Entry Title from DB', 'entry_text': 'Text From DB'})
def update(self, entryID, titleFromPOSTFormInput, textFromPOSTFormInput):
# Update DB with passed in arguments. entryID comes from URL,
# other two entries come from POST
cherrypy.response.headers['Content-Type'] = 'application/json'
return json.dumps({'success':True})
# Setup URL routes
d = cherrypy.dispatch.RoutesDispatcher()
d.connect(name='blog_entry', route='blog/:entryID', action='get',
controller=Blog_Controller(),
conditions={'method': ['GET']})
d.connect(name='blog_entry', route='blog/update/:entryID', action='update',
controller=Blog_Controller(),
conditions={'method': ['POST']})
config = { '/' : { 'request.dispatch': d } }
cherrypy.tree.mount(root=None, config=config)
# Start the webserver
engine = cherrypy.engine
try:
engine.start()
except:
sys.exit(1)
else:
engine.block()

It sounds like this might be a good use case for GWT with Restlet.

Forgive me for tooting my own horn, but if you are doing AJAX REST stuff with jQuery, you should probably check out my JSON-REST plugin:
http://plugins.jquery.com/project/rest
If you are getting XML back, then this won't be as useful, but you may still be able to adapt some of the code to your needs.