We Keep Coding

Asynchronous Message-Passing and Microservices

I am planning the develop of a microservice based architecture application and I decided to use kafka for the internal communicaton while I was reading the book Microservice Architecture by Ronnie Mitra; Matt McLarty; Mike Amundsen; Irakli Nadareishvili where they said:
letting microservices directly interact with message brokers (such as
RabbitMQ, etc.) is rarely a good idea. If two microservices are
directly communicating via a message-queue channel, they are sharing a
data space (the channel) and we have already talked, at length, about
the evils of two microservices sharing a data space. Instead, what we
can do is encapsulate message-passing behind an independent
microservice that can provide message-passing capability, in a loosely
coupled way, to all interested microservices.
I am using Netflix Eureka for Service registration and discovery, Zuul as edge server and Hystrix.
Said so, in practice, how can I implement that kind of microservice? How can I make my microservices indipendent from the communcation channel ( in this case Kafka)?
Actually I'm directly interacting with the channel, so I don't have an extra layer between my publishers/subscribers and kafka.
UPDATE 06/02/2018
to be more precise, we have a couple of microservices: one is publishing news on a topic (activemq, kafka...) and the other microservice is subscribed on that topic and doing some operations on the messages that are coming through. So we have these services that are coupled to the message broker (to the channel)... we have the the message broker's apis "embedeed" on our code and for example, if we want to change the message broker we have to change all the microservices that made use of the message broker's api. So, they are suggesting to use a microservice(in the picture I assume is the Events Hub) that is the "dispatcher" of the various messages. In this way it is the only component that interacts with the channel.

A general foreword - Don't do it if you don't need it. Introducing a queue system can be a big improvement if you are dealing with high number of events and events backing up issues etc. But if you don't face any issues you are probably better off with the lower complexity of a direct service communication.
Back to your question - It sounds like you want to abstract your communication with the queue because you are worried about the effort for replacing the queue with a different system - Is that correct?
In this case you can either do what you proposed - Develop a new service in the middle. This comes with all the baggage of a physical service (including deployment, scaling, etc).
Or the second alternative is to write a client library that abstracts the queue the way you want and allows you to reuse it in all services requiring to participate in the queue. This way you don't have to physically deploy another service for this purpose but you are still in full control of what your interface to the queue should look like and you have a single piece of code to incorporate changes (at least toward the direction of the queue). This would work given you are sure the app-facing side of the library can be stable enough.
But, again, don't do any of those in the first iteration when you are not sure you need all the complexity. (Over-engineering is a dangerous thing)

You should create a Interface lets say "Queue" which provide all functionalities which you want from Kafka or RabbitMQ, the create diff. impl like KafkaQueue and RabbitMQQueue of the Queue interface and inject the right impl which you want to use in your system.
In this your if new queue system is used , your existing code will not be changed
Creating another microservice is an extra overhead in this case

In a service architecture proper way to make your code independent out of constraints of communication channel is by having properly modeled self-sufficient messages. Historic examples would be WSDL in document mode, EDIFACT, HATEOAS etc. From this point of view microservices with spring-boot and kafka are just different implementation of same old thing done since mainframes ruled the world.
Essentially if you take a view of your app as blackbox asynchronous server; everything app does is receives events and produces new ones. It should not matter how events are raised within app. Http requests, xml within jms messages, json in kafka, whatever - all those things are just a way to pass events and business layer of application should respond only to a content of events.
So business layer is usually structured around some custom model/domain which are delivered as payload. Business layer is invoked/triggered by listener/producer layer which talks to communcation channel (kafka listener, http listener etc..). Aside from logging and enforcing security you should not have communication channel logic in app. I have seen unfortunate examples of business logic driven by by originating jms connection or parsing url of request. If you ever have this in your code you have failed to properly structure your code.
However that is easier to say than to implement. Some people are good at this level of modeling, and some never learn.
And there is no other way to learn but to try and fail.

Criteria to select b/w soap and restful ws and why rest is cacheable?

I have read lot a whats are difference b/w soap and restful web services. I have got specific doubts for which i did not get answers. Here they are :-
What goes in favor of Restful web services
In nutshell everybody seems to be preferring Restful web services over soap. main reason behind it easy to develop and understand. Also faster mainly because of light weight data exchange format like json. Also in Performance in restful web services is better because less data is traveled over network(Soap involves extra layer of saop message under http request). So far so good.
Accepted answer at how is Restful web services better than SOAP based webservices also says
REST naturally fits for Web/Cloud API's, whilst SOAP fits for
distributed computing scenarios.
I did not get whats the difference between Web and distributed computing scenarios. Web is also case of distributed computing scenarios. Is n't it ? so how come one is better for web while another is for distributed scenarios ? (Q1)
What goes in favor of SOAP :-
Same answer also says that SOAP caters for stateful operations. As per my understanding its not true. If you need to maintain the state you need to maintain thru you code like sending some unique ID in both request/response that relates them Is n't it? (Q2) If thats the case that can be done in restful web services also.
Some says SOAP is better in terms of security. I don't what security soap provides that rest does not? (Q3)
Soap is probably is better in one sense that it has WSDL(that too generated by tools) doc thru which clients can generate their respective stubs. In restful webservices developer has to create the comprehensive doc himself so that client knows about the input request parameters. correct ? (Q4)
Note :- I have referred Q for question(Q1 is Question1)

There are 3 reasons to prefer REST over SOAP
Resources have URLS that identify them. If you want to share the result of some API operation with a friend, then you don't have to instruct them verbally on which API method to call and which parameters to pass. Instead, you can just share with him the URL. (for safe operations only). A great example is restaurant sites that are built in flash. If you just want to share with a friend the menu of the restaurant, you can't do that very easily. You can only tell him which buttons to press when the flash page loads.
With REST you can take advantage of the existing HTTP infrastructure on the internet to do a lot of work, i.e. caching, resource conflict management, and so on. You don't have to reinvent the wheel.
Related to #2, many developers are already used to working with REST architectures. If you use REST you significantly cut down on the learning curve that new devlopers will have to incur when learning how to use your service.

REST is cache-able for GET request. For SOAP, when using HTTP as the transfer mechanism, is sent via HTTP POST requests which are not cache-able.

what technology to use for a database service?

I am going to make a small trade management system. I want to make a independent database service to which all the other client connect. The database will be MYSQL and I will be using Java for making the service. The client can either be a Web Application or a desktop application using Java Swing (has not decided yet). There will be another layer sit between the client and the database service to handle the business logic (I call it trade service).
The architecture is something like: Client -> Trade Service -> Database Service.
My questions is that what client/service communication technology is the best suitable one for client->Trade Service and the best suitable one for Trade service -> Database.
Shall I make it s RESTful service? SOAP? Using RPC? Or any other technologies?
Many thanks for your help. Any idea or suggestions are welcome.

Take a look at RabbitMQ, A pool messaging service
http://www.rabbitmq.com/
It's Robust, flexible, fast and scalable and you can use it to communicate in Java, PHP, or whatever technologies ou want.

Shall I make it s RESTful service? SOAP? Using RPC?
These are all very similar approaches in that they are over HTTP so - assuming that's what you want; I would recommend using RESTful. You'll have lot's of examples to work with and it will allow you flexibility in the future to do things like switching out the UI layer for a smart phone app or desktop app.
Regardless of what model you pick you should understand how it works first and build in things like security and guidelines early. Do your homework now. Trying to change the middle layer of a design like this is a pain.

There is no blanket answer to your question, there are instead options based on your skill set. Do you conceptually understand the HTTP spec completely and be able to extend it to REST, that works very closely with HTTP (common creation ancestor). Do you better understand the traditional method invocation of SOAP? Are you tied in your ecosystem to a specific language, as this can impact which tools you choose from.
If you were paying me to write a service based on the simple requirements you have given (which is nearly impossible), I would create a domain driven design service (your business layer) with a RESTful interface and Spring JDBC for data access. That is me, and what I work in most often. My partner in crime at work would probably choose SOAP and Hibernate.

I think what you're taking about is Queues, and I'm guessing you need a managed service for that. Queues can be the glue between your micro-services. Some of the vendors I know which have Queue as a Service are :
CloudBoost.io : https://www.cloudboost.io
Check out https://tutorials.cloudboost.io/en/queues/basicqueues for documentation.
Iron.io : https://www.iron.io
P.S : I work at CloudBoost

Scalable automatic email classification service

We're currently working on an application that enable the user to register with one or more email account so that his emails could be automatically classified. The front-end has been implement using Ruby, however, the back-end (email classifier) is written in java and uses the WEKA API. The question is how could we integrate the front-end (Web interface written in Ruby) with the back-end (email classifier written in java) in a scalable way (handling large number of users simultaneously..

I am not sure what is an email classifier. But in any similar problem, the best solution I recommend creating a RESTful API for your java service. This can be done very elegantly with the right tools. The API should be over HTTP which returns JSON. Use a library like Jackson that serialize to JSON.
On the ruby side, you an easily parse that JSON and deserialize.
This is a very scalable solution because HTTP calls are stateless and already scalable. Thread is used and thrown away. If you need more power, then just add more machines.
The Rails app can also start caching some calls. But that is premature optimization.
If there is no logic and only a common database, then just share that common database between the two apps. But it sounds like the Java app needs to do some work. This is a common approach with APIs. It also doesn't limit you to Ruby. You can create JSONP service for AJAX or any other client that can understand JSON.

If you want a new email alert just reverse which RESTful API you are exposing. Instead of exposing the Java app as a RESTful API, expose the Rails app API. For example /user/ID/newmail.
The Java app would then call the Rails app when a new email arrives.
Btw:
How did you implement a scalable system in Java for checking 1000s of email accounts?

As the amount of data you're using to train the classifier with grows, you may find that you might want to use ensemble algorithms (where a group of n nodes form the ensemble) and split the training data up over each of the n nodes.
To classify a new datapoint, you can use a voting system where each of the n nodes gets to "vote" on what the new datapoint should be classified as. The classification with the most votes wins.

Usage of a Data Access Layer (DAL) in a SOA design

As we are in the beginning phases of rejuvenating our application in to SOA design I have some questions that I can not get a clear answer/picture on.
I have been doing a lot of reading, mostly around books from Thomas Erl and following that design pattern of understanding what Task Services, Entity Services and Utility Services are.
What I am stumbling on is the whole DAL concept of how that would look. So this is more of a verification of understanding or a clarification so as to help make the best approach for our platform.
So background. We currently have several web based e-commerce applications that have been pretty much been built in silos and are again pretty much a copy of each other. We have supporting applications such as Daemons and misc web services out there. Many of these applications are older then 5 years and are build on only technology (Model 1). All of our applications are centered around conducting auction sales. So during a sale event we will be taking bids from users, determine who is winning and display that information back. Each sale event has a set amount of time that they will be available to the users.
The company is moving towards a SOA solution as a lot of things we end up doing can be shared across not only our group but across other groups.
So what I understand on the DAL is that it in itself is a service which will sit on top of Data, in this case different Databases - MSSQL, ORACLE, MSSQL. Each of these databases have different schema's (Oracle) etc.
So the services (Task, Entity, Utility and Presentation Tier if needed) will make calls to the DAL to retrieve data. It is the responsibility of the DAL to know, from the contents of the message to determine what it needs to do in order to fulfill the request.
So for example, we have a Security Service candidate. This service needs to authenticate with LDAP and to authorize from the data that is stored for that given application.
The thought here is that a Utility service will be created to wrap up all the operations required to communicate with LDAP and that the Security Service will call upon the Utility Service and to the DAL to fetch the authorization data. The DAL then has the responsibility to go to the correct database/schema to retrieve the information. The information will be in XML format (standard SOA communication).
So, am I on the right track here? Have others done similar things or not? What other things do I need to consider (Currently getting the statistics on how many bids we take in an hour - on average).
Should each service have its own DAL - for example should the Security Service have the DAL as part of the service or should DAL be a shared service in which all services can use?

In your case, the approach to use for a full SOA based deployment would be to use an ESB, Identity provider and a data services solution.
To break it down, the DAL should be implemented using data services, in this way, this service will be a globally accessibly service in a language neutral way, and will support re-use and loose coupling. So all your data access logic can be implemented as web service operations in a data service.
So for the authentication and authorization management, in the SOA world, there's a standard called XACML, which is used for fine grained authorization management. So what you will need is an XACML server, who would authorize the user according to a specific criteria, where this should also have the ability to authenticate with LDAP.
Then your "Security Service" will be implemented in a service at the ESB, where that service will query the identity provider for authentication/authorization and according it's response, it will call the appropriate operations in the data service, with suitable parameters to fetch the data, and return it to the user.
The above scenarios can be implemented using WSO2 Data Services Server, WSO2 Identity Server and WSO2 ESB respectively, which are open source products, and can be freely used and found here.

i once worked with (developing) an soa project that used a "data service". it was some time ago, and i was only involved marginally, but my recollection was that it ended up being too complicated and slow.
in particular we had no real need for a data service - it would have made more sense to have placed the same abstractions in a library layer, which would have given better efficiency and no real loss of functionality (for our particular needs). this was exacerbated by the fact that the data tended to be requested in many small "chunks".
i guess it comes down to the trade-offs involved in the implementation. in our case, with a relatively closed system and a single underlying database technology, we could have easily exploited the support for distributed access that the database provided; instead we ended up duplicating this in a slower, more general, message bus, which added nothing except complexity. but i can easily imagine different cases where access to data is more "distant".

How you have to use SOA for your design is depends on the its requirements.
In generally you can write coarse grain services and expose them as web services. In your case you can write some services which calls the databases and produce the results. In this case authorization logic can also be written with the service logic.
The other approach is to use an ESB or BPEL engine to write the integration logic and expose the integrated service as a web service. In this case you can use some data services to expose data base data in xml format and integrate them. You can use services for different sachems and call the correct service with the request data. And the authorization logic can also be added to service integration logic.
Security aspects such as authentication, confidentiality, integrity is considered as non functional requirements and hence can be engaged to any service without writing an explicit security service.
Following articles describes such sample possible integration of services as mentioned in the second approach.
http://wso2.org/library/articles/2011/05/integrate-business-rules-bpel
http://wso2.org/library/articles/2011/06/securing-web-service-integration