We Keep Coding

Which HTTP verb to use to trigger publication of Kafka messages

Background
I need to implement a way for developers in our team and the product owner to be able to trigger the re publication of some Kafka messages after querying the DB for our Spring Boot application. DB will be queried for the data but no modification will be performed. So the action has no effect on the resource held by the application.
Potential Solution
I am thinking about implementing a REST endpoint to achieve this. I know that this is not a good use case to be implemented via REST as there is no "State Tranfer" for the resources held by the application. So...
Question
Any other recommendations for alternative ways to achieve this apart from REST endpoint ?
If I were implement it via a REST endpoint which HTTP verb should I use ?
( I have checked and haven't found any answered question of this nature in any previous SO post. )

Which HTTP verb to use to trigger publication of Kafka messages?
You answer this question by reviewing the semantics of the HTTP methods, and deciding which is appropriate to your use case.
For this example, the reasonable candidates are GET and POST. (Technically, you could consider HEAD, which is sort of a specialized case of GET -- I'll ignore it here).
The important distinction to understand is that GET semantics include safe; the description is "effectively read only", but the important bits are the implications of that.
The purpose of distinguishing between safe and unsafe methods is to allow automated retrieval processes (spiders) and cache performance optimization (pre-fetching) to work without fear of causing harm.
Automated retrieval means that you may see GET requests even when there is no business motivation for them. If that's OK, then using GET is fine -- and it even has some advantages when the requests are being sent across an unreliable network.
On the other hand, if republishing is "expensive" or should otherwise only be done "on purpose", then POST is a more appropriate choice.
It might help to imagine this in the context of a web page in a browser. If you provide access via an a element, then the browser is allowed to try to optimize the user experience by pre-fetching the resource. On the other hand, using a form element with
method: POST won't be pre-fetched, because compliant browsers are aware that the request isn't expected to be safe.
Online opinion about [REST] seems to be divided.
REST suffers a lot from what Martin Fowler called Semantic Diffusion.
Semantic diffusion occurs when you have a word that is coined a person or group, often with a pretty good definition, but then gets spread through the wider community in a way that weakens that definition. This weakening risks losing the definition entirely - and with it any usefulness to the term.
REST is an architectural style; the reference application built using that style is the World Wide Web.
The only other open question which remains is, whether REST is the right way to trigger a job
Depends on the constraints you are operating under, and how ambitious you are being. "Everybody" has web clients available to them, so it's a cheap way to transport a message from client to server. On the other hand, HTTP requests are not micro small, and there's lots of capability built in that may be no more than drag in your use case.
Horses for courses.

Understanding ESB [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
though I understand what system integration is, I am a bit new to all the newest approaches. I am farily familiar with web services and JMS but I feel utterly confused by the concept of an ESB.
I have done some research but I still don't really get it. I work much better by example rather than theory.
So can someone please illustrate a simplistic example to demonstrate why one would use an Enterprise Service Bus vs just a Queue , a web service , the file system or else?
I would like the example to amplify the capabilities of the ESB which could not be achieved by any other conventional intgration method or at least not with the same efficiency.
All replies are greatly appreciated.
Thanks,
Bob

This is going to sound a bit harsh, but basically if you needed an ESB, you'd know you needed an ESB.
For a majority of use cases, the ESB is a solution looking for a problem. It's a stack of software over engineered for most scenarios. Most folks simply do not do enough variety of processing to warrant it. The "E" for "Enterprise" is notable here.
In a simple case:
tail -F server.log | grep SEVERE >> severe.log
THAT is a trivial example of an instance of an ESB scenario.
"But that's just a UNIX command pipeline!"
Yes, exactly.
The "ESB" part is the "|" and the ">>"
The ESB is the run time within which you can link together modules, monitor traffic, design all sorts of whacky scenarios like fan outs and joins, etc. etc.
ESBs are notable for having a bunch of connectors to read a bunch of sources and write a bunch of destinations. They're notable for weaving more complicated graphs and workflows for processing using rather coarse logic blocks.
But what most folks typically do is:
input -> DO_STUFF -> output
With an ESB they get:
ESB[input -> DO_STUFF -> output]
In the wild, most pipelines simply are not that complicated. They tend to have one off logic that's not reusable, and folks tend to glob it together in to a single logic module.
Well, heck, you can do that with a Perl script.
Long pipelines in ESBs tend to be more inefficient than not. With lots of marshaling of data in to and out of generic modules (since you rarely use a binary payload).
So, say, CSV comes in, converts to XML, process it, output XML for input to another step as XML, which marshals it, works on it, converts it back in to XML for Yet Another step. Rinse and repeat until the CPU hits 400% (multi-core FTW).
Then some one comes up with "Hey, if I drag an drop these modules together in to a single routine, we skip all this XML junk!", and you end up with "input -> DO_STUFF -> output".
For large systems, with lots of web services that need to do casual, ad hoc integration, they can be fine. If you're in a business that does that a lot, they can work really well. When you have dozens of pipelines, they can help with the operational aspect of managing them.
But for complicated pipelines, if you have a lot of steps, maybe it's not such a good idea beyond prototyping, especially if there's any real volume involved. Mind, you may not having any choice, depends on the systems you're integrating.
If not, if you have a single interface you need to stand up -- then just do it. Do it in Perl, in Java, in C#, in whatever. Don't run out and spool up some odd 100MBs of infrastructure and complexity that you now to get to learn, master, and maintain.
So, again, if you needed an ESB, you'd know it. Really. You'd have whatever system you've built together of disparate stuff, you'd be fighting it, talking to colleagues about what a pain all this stuff is, and you'd stumble across some link to some site to some vendor and read a white paper and go "THAT'S IT!", but if you haven't done that yet, then you're not missing anything.

ESB is for the cases where you do have that web service and queue and file system all in the same system and need to integrate them.
An ESB product usually solves the following
Security
Message routing
Orchestration (which is advanced message routing)
Protocol transformation
Message transformation
Monitoring
Eventing
You can do all of these with other tools as well and if you just need one or two of these capabilities you can probably do without and ESB (as it introduced additional complexity) but when you need several of them an integrated solution in the form of an ESB can be a better solution.

As #WillHartung concluded, ESBs tend to be properly used in large, complex situations. And that's why it's named Enterprise Service Bus.
Now, to actually answer your question, ESBs typically:
Communicate over several protocols (e.g. HTTP, Message Queue, etc.), for both input and output
Establish a common message format, and often translate from other formats into the 'canonical' format
Provide endpoint transparency (e.g. you send a message to the bus, and get an answer back, but you don't explicitly know what service, also connected to the bus, handled your request.
Provide monitoring and management capabilities
Facilitate versioning of services and messages
Enforce security, when needed.
So, as you can see, it's for when doing a lot of point-to-point communication ("just do it") would be a huge, unmanageable pile of spaghetti. Indeed, most places that I've seen SOA implemented, it's replacing that huge pile of spaghetti that already exists.

An ESB is an enterprise service bus, an infrastructure backplane if you like for a service-oriented architecture. Imagine the chaos of hundreds of services happily reusing each other. How do manage such an environment? How do you provide flexible, decoupled routing between your services? How do you avoid point-to-point spaghetti architecture? How do you manage transactions and security across a hybrid technology landscape? How do you track where messages are in complex flows across multiple systems?
You use an ESB.
ESBs typically allow you to design flows across multiple systems in an XML configuration language, offering you a host of EIS adaptors, transformation and mediation plugins etc. Some will offer an IDE to help you design flows. Some ESBs are very expensive, some are open source.
If you want to get a feel for ESB, check out either Mule or WSO2, both good open source products, or even Spring Integration which is a non-clustered solution but excellent for decoupling Java from the underlying external interface points.

Real life experience with the Axon Framework [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
As part of researching CQRS for use with a project, I ran across the Axon Framework, and I was wondering if anyone has any real life experience with it. Just to be clear, I'm asking about the framework, not CQRS as an architectural pattern.
My project already uses Spring and Spring Integration which fits nicely with Axon's own requirements, but before i dedicate a lot of time to it, I would like to know if anyone has some first hand experience. In particular I'm interested i possible pitfalls that are not immediately apparent from the documentation.

The framework relies heavily on eventsourcing, which means that all state changes are >written to the data store as events. "
This is completely untrue, it does not rely heavily on event-sourcing. One of the implementations for storing the aggregate in this framework use Event-Sourcing but you can easily use also the classes provided to use a standard relational model.
It is just better with event-sourcing.
So you have a historical reference of all your data. This is nice but makes changing your >domain after you've gone in production a very daunting proposition especially if you sold >the customer on the system's "strong auditability" "
I don't think it is a lot easier with a standard relational model that only stores the current state.
The framework encourages denormalizing your data, to the point that some have suggested >having a table per view in the application. This makes your application extremely >difficult to maintain, especially when the original developers are gone"
This is unrelated to the framework but to the architectural pattern in use (CQRS).
And sorry to mention that but having one denormalizer/view is a good idea as it stays a simple object.
So maintenance is easy because SQL request/insertion as also easy.
So this argument is not very strong.
How about a view which uses a 1000 tables model with inner joins everywhere and complex SQL queries?
Again, CQRS helps because, basically, the view data is just a SELECT * from the table which correspond to the view.
if somehow you made a mistake in one of the eventhandlers, your only option is to >"replay" the eventlog, which depending on the size of your data can take a very long >time. The tooling for this however is non-existent.
I agree on the point that currently there is a lack of tooling to replay events and that this can take a long time. However, it is theoretically possible to only replay a portion of the event and not all the content of the event store.
Replaying can have side effects, so >developers become scared of doing it
Replaying event have side effects -> that's untrue. For me side effects means modifying the state of the system. In an event-sourced CQRS application, the state is stored in the event-store. Replaying the events does not modify the event store.
You can have side effect on the query side of the model yes. But you don't care if you have made a mistake because you are still able to correct it and replay the event once again.
it's extremely easy to have developers mess up using this framework. if they don't store >changes to domain objects in events, next time you replay your events you are in for a >surprise.
Well if you misused and misunderstand the architecture, the concept, etc. then ok I agree with you. But perhaps the problem is not the framework here.
Should you store delta's ? absolute values ? if you don't keep tabs on your developers >you are bound to end up with both and you will be f***ed
I can say that for every system I would say that it's unrelated directly to the framework itself. It's like saying, "Java is crap because you can messed up everything if someone codes a bad implementation of hashCode and equals methods."
And for the last part of your comment, I already seen samples like helloWorld with the Spring framework.
Of course it is completely useless in a simple example.
Be careful in your comment to make a difference between the concept (CQRS + EventSourcing) and the framework. Make a difference please.

Since you have stated that you want to use CQRS for your project (and I assume that the JVM is your target platform) I think Axon Framework is an excellent choice.
I have built a fairly complex trading platform on it (no, the trading sample is not complex) and I have not seen any obvious flaws of the framework.
Since I use EventSourcing, the test fixtures made it very easy to write BDD style "given, when, then" tests. This lets you treat an aggregate as a black box and concentrate on checking that the correct set of events come out when you put in a certain command.
About pitfalls: before jumping in, make sure
That you have the concepts of CQRS figured out.
Make a list (paper, whiteboard, whatever) of all your aggregates, command handlers, event handlers, sagas, commands and events. This is the hard part of building your system, figuring out what it should do and how. After this, the reference manual should show you how to wire it all together with Axon.
Some non Axon specific points:
Being able to rebuild the view store from events is a concept of EventSourcing, and not something that is exclusive to Axon, but I found it pretty easy to create a service that will send me all events from an aggregate type, aggregate id or a certain event type.
Being able to build a new reporting component one year after the project is launched and instantly get reports on data from the time of the project launch and onwards is awesome.

I've been using AxonFramework for more than one year on a complex project developed for a big bank.
The requirements were demanding, customer's expectations were high, and release times narrow.
I've choosed AxonFramework because, at the project kick off moment, it was the most complete and the best documented implementation of CQRS available in Java, well designed, easy to integrate, to test and to extend.
After more than one year I think that these considerations are still valid and current.
Another consideration has guided my choice: I wanted that the commitment on such a difficult project to become a training opportunity for me and other members of the team.
We started to develop with AxonFramework version 1.0 and moved to version 1.4 as newer versions were released.
Our team experience with CQRS and the implementation provided by the AxonFramework was absolutely positive.
It provided us with a consistent and uniform manner to develop each feature that guided us and make you feel at ease.
Without it some features of the application would have been much more complicated to develop.
I am referring mainly to the various long-running processes that need to be handled and to the related compensation logic, but also to the many business logics pieces that have been necessary, here and there, that fitted nicely and uncoupled in the event driven architecture promoted by CQRS.
Our choice was to be conservative in the write model, so we preferred a JPA based persistence instead of the event sourced one.
The query model is made up of views. We have tried to make sure that each view contains all the required data from a single page using intermediate views when necessary.
Anyhow we developed the write model as we were applying event sourcing, so we take care of modifying the state of aggregates exclusively through events. When the customer asked for a cloning function of a very complex aggregate it was just a matter of replaying the source events (with uuid translated) to a brand new instance - the down side in this case have been the events upcasting (but this functionality was greatly improved in the imminent 2.0 version).
As in each project during the development we found a lot of bugs, in our code mainly, but also in components supposed to be mature and stable, like the application server, the IoC container, the cache, the workflow engine and some of the other libraries that are easily to be found in any large J2EE application.
As any other human product AxonFramework was not immune to bugs too, but surprisingly for a young and niche project like this, they have been few, not critical, and quickly resolved by new releases.
The kind and immediate support provided by the author on the mailing list is another invaluable feature and helped me a lot when I was in trouble.
The application was released in production a year ago and is currently maintained and under active development of new features.
The customer is satisfied and asks for more.
When to use AxonFramework is more a matter of when to use CQRS. For a response it's worth to go back to the official documentation: http://www.axonframework.org/docs/1.4/introduction.html#d4e51
In our case definitively it was worth it.

The OP specifically asks about the pitfalls relating to the Axon Framework rather than CQRS. This makes the question difficult to answer, as Axon started as a fairly faithful implementation of the famous book by Eric Evans
The main advantage is that it does exactly what it says on the tin: it handles the hard parts of a CQRS based design for you: aggregates, sagas, event sourcing, command handlers, event handlers, BASE consistency etc. When you follow the best practices, you end up with a highly responsive and horizontally scalable application. If you use it with event sourcing, your data is completely auditable, and at least in theory, you can determine the state your application had at any given point in time. Tooling to do this is not provided; you will have to roll your own.
The main developer of the framework is very approachable and extremely knowledgeable on the subject of high performance and scalable computing in java. He tends to answer every question on the mailing list within a few hours. This is both an advantage and the major pitfall: at this time (early 2014), the Axon Framework depends heavily on one person. The rest of the pitfalls I would like to mention are probably more the result of event sourcing than of CQRS or Axon (as of 2018 the framework is supported by the company Axoniq)
Design your data model very carefully upfront. Though it is easy to add to, making fundamental changes to your datamodel can be very difficult. If you make a fundamental mistake in the datamodel, your application may not perform well, or even fail to work at all. For example, if you choose a tree shaped data model, with one long lived aggregate root at the top, this aggregate may grow very large as it accrues more and more events over time, and it may take a long time to load and store. I don't know what will happen if this goes on until an instance of the aggregate no longer fits in RAM, but I imagine could be bad. Don't do it that way.
Another pitfall (event sourcing related) is that, after a number of revisions, it can become increasingly difficult to reason about the state of an aggregate, as you sometimes have to keep in mind not only what the code does today, but also what it did in the past. This definitely makes replaying (a portion of) the event store to rebuild a view table a non trivial task.
Fixing data errors can be more difficult than with a 'traditional' design. Rather than a simple SQL statement, you will often need to make a command to change the state of your application. If the error in your data was caused by a faulty event handler, you can usually just fix the bug, clear the snapshots and let he events for the aggregate be replayed. If your bug caused spurious events to be applied, it can me much more trouble to fix. The faulty events will stay in the event store, and you may have to apply some new ones to restore your data to the correct state, or change the code to ignore or fix their behaviour.

While the framework itself is written decent enough, using it in a real world project has been nothing short of a nightmare and the choice of this framework imo was a major contributing factor to this project's failing.
The framework relies heavily on eventsourcing, which means that all state changes are written to the data store as events. So you have a historical reference of all your data. This is nice but makes changing your domain after you've gone in production a very daunting proposition especially if you sold the customer on the system's "strong auditability"
You cannot have ops guys make ad-hoc changes to the database
The framework encourages denormalizing your data, to the point that some have suggested having a table per view in the application. This makes your application extremely difficult to maintain, especially when the original developers are gone
if somehow you made a mistake in one of the eventhandlers, your only option is to "replay" the eventlog, which depending on the size of your data can take a very long time. The tooling for this however is non-existent. Replaying can have side effects, so developers become scared of doing it
it's extremely easy to have developers mess up using this framework. if they don't store changes to domain objects in events, next time you replay your events you are in for a surprise. Should you store delta's ? absolute values ? if you don't keep tabs on your developers you are bound to end up with both and you will be f***ed
There is practically no adoption of this framework, so googling for answers will not do you any good
Even though the framework does not yet support distribution it's written with it in mind and the api's are a pain to work with because of it. Firing off an event is async by default and if you want to check if an exception was raised executing the command, say a duplicate username exception, you need to pass in a listener to your commandhandler which is a future, then you wait for the future's result to come in, handle any checked exceptions, interuptedexception etc and then you can grab the exception that was thrown from the future. Ofcourse which exceptions a command can raise is not apparent from the api. Defeating the purpose of checked exceptions
Check out some of the example apps. I somehow need a unit of work listener to create an addressbook application? My goodness...

I am currently with a team working on an online casino platform launching our brand Casumo this summer. The domain and platform is build using Axon Framework and so far it it has served us solidly.
A lot of time has been saved not having to build all the infrastructure needed for command handling, event routing, event sourcing, snapshoting etc and the APIs are really nice to work with. The one bug we found in the framework so far was fixed in .. release 12 hours later and Allard is always quick to take suggestions on new features and discussing ways to leverage the framework to fulfill your needs.

socket -V- rest performance

I have done some searching but haven't come up with anything on this topic. I was wondering if anyone has ever compared (to some degree) the performance difference between an RPC over a socket and a REST web service. If both do the same thing, which would have a tendency to be the better performer? I've already started building some socket code and would like to know if REST would give better performance before I progress much further. Any input would be really appreciated. Thanks indeed

RMI
Feels like a local API, much like
XMLRPC
Can provide some fairly nice remote
exception data
Java specific means this causes lock
in and limits your options
Has horrible versioning problems
between different versions of clients
Skeleton files must be compiled in
like CORBA, which is not very flexible
REST:
easy to route around firewalls
useful for uploading files as it can
be rather lightweight
very simple if you just want to shove
simple things at something and get
back an integer (like for uploaders)
easy to proxy security behind Apache
and let it take the heat
does not define any standard format
for the way the data is being
exchanged (could be JSON, YAML 1.0,
YAML 2.0, arbitrary XML format, etc)
does not define any convention about
having remote faults sent back to the
caller, integer codes are frequently
used, but method of sending back data
is not defined. Ideally this would be
standardized.
may require a lot of work on the
client side caller of the library to
make use of data (custom serialization
and so forth)
In short from here
web services do allow a loosely
coupled architecture. With RMI, you
have to make sure that the objects
stay in sync in all applications
RMI works best for smaller
applications, that are not
internet-related and thus not scalable

Its hard to imagine that REST is faster than a simple socket connection given it also goes over a Socket.
However REST may be performant enough, standard and easier to use. I would test whether REST is fast enough and meets your requirements first (or one of the many other existing solutions) before attempting your own Socket solution.

Has anyone migrated from Struts 1 to another web framework?

On my current project, we've been using Struts 1 for the last few years, and ... ahem ... Struts is showing its age. We're slowly migrating our front-end code to an Ajax client that consumes XML from the servers. I'm wondering if any of you have migrated a legacy Struts application to a different framework, and what challenges you faced in doing so.

Sure. Moving from Struts to an AJAX framework is a very liberating experience. (Though we used JSON rather than XML. Much easier to parse.) However, you need to be aware that it's effectively a full rewrite of your application.
Instead of the classic Database/JSP/Actions scheme for MVC, you'll find yourself moving to a Servlet/Javascript scheme whereby the model is represented by HTTP GET requests, actions are represented by POST/PUT/DELETE requests, and the view is rendered on the fly by the web browser. This leads to interesting challenges in each area:
Server Side - On the server side you will need to develop a standard for exposing data to the client. The simplest and easiest method is to adopt a REST methodology that best matches your data's hierarchy. This is fairly simple to implement with servlets, but Sun also has developed a Java 1.6 scheme using attributes that looks pretty cool.
Another aspect of the server side is to choose a transmission protocol. I know you mentioned XML already, but you might want to reconsider. XML parsers vary greatly between browsers. One browser might make the document root the first child, another one might add a special content object, and they all parse whitespace differently. Even worse, the normalize() function doesn't seem to be correctly implemented by the major browsers. Which means that XML parsing is liable to be full of hacks.
JSON is much easier to parse and more consistent in its results. Javascript and Actionscript (Flash) can both translate JSON directly to objects. This makes accessing the data a simple matter of x.y or x[y]. There are also plenty of APIs to handle JSON in every language imaginable. Because it's so easy to parse, it's almost supported BETTER than XML!
Client Side - The first issue you're going to run into is the fact that no one understands how to write Javascript. ESPECIALLY those who think they do. If you have any books on Javascript, throw them out the window NOW. There are practically no good books on the language as they all follow the same "hacking" pattern without really diving into what they are doing.
From the lowest level, your team is going to need remedial training on Javascript development. Start with the Javascript Client Guide. It's the de facto source of information on the language. The next stop is Douglas Crockford's videos on Javascript. I don't agree with everything he has to say, but he's one of the few experts on the language.
Once you've got that down, consider what frameworks, if any, you want to use. Generally speaking, I dislike stuff like Prototype and Mootools. They tend to take a simple problem and make it worse. None the less, you can feel free to evaluate these tools and decide if they'll work for you.
If you absolutely feel that you cannot live without a framework because your team is too inexperienced, then GWT might fit the bill. GWT allows you to quickly write DHTML web apps in Java code, then compile them to Javascript. The PROBLEM is that you're giving up massive amounts of flexibility by doing this. The Javascript language is far more powerful than GWT exposes. However, GWT does let Java developers get up to speed faster. So pick your battles.
Those are the key areas I can think of. I can say that you'll heave a sigh of relief once you get struts out of your application. It can be a bit of a beast. Especially if you've had inexperienced developers working on your Struts model. :-)
Any questions?
Edit 1: I forgot to add that your team should study the W3C specs religiously. These are the APIs available to you in modern browsers. If you catch anyone using the DOM 0 APIs (e.g. document.forms['myform'].blah.value instead of document.getElementById("blah").value) force them to transcribe the entire DOM 1 specification until they understand it top to bottom.
Edit 2: Another key issue to consider is how to document your fancy new AJAX application. REST style interfaces lend themselves well to being documented in a Wiki. What I did was a had a top level page that listed each of the services and a description. By clicking on the service path, you would be taken to a document with detailed information on each of the sub-paths. In theory, this scheme can document as deep as you need the tree to go.
If you go with JSON, you will need to develop a scheme to document the objects. I just listed out the possible properties in the Wiki as documentation. That works well for simple object trees, but can get complex with larger, more sophisticated objects. You can consider supplementing with something like IDL or WebIDL in that case. (Can't be much worse than XML DTDs and Schemas. ;-))
The DHTML code is a bit more classical in its documentation. You can use a tool like JSDoc to create JavaDoc-style documentation. There's just one caveat. Javascript code does not lend itself well to being documented in-code. If for no other reason that the fact that it bloats the download. However, you may find yourself regularly writing code that operates as a cohesive object, but is not coded behind the scenes as such an object. Thus the best solution is to create JSDoc skeleton files that represent and document the Javascript objects.
If you're using GWT, documentation should be a no-brainer.

Check out the Stripes Framework. If you are familiar with struts then stripes will make sense to you, but it's so much better. They have a Stripes vs Struts section on their website. You could check that out and see if it interests you. It allows you to work with any ajax framework you want, and I don't think it would take long to migrate from struts to stripes.