I use several systems connected by Service oriented architecture, those systems include : .NET technology, Java WebLogic Service, and SAP RFC.
is it possible to achieve distributed transaction across these different technologies ? for example when updating SAP failed, we need to make sure that .NET, and Java transaction won't happen at all.
Really appreciate your input, or maybe you can point out where we can learn the basics how to do it?
I would do this as follows:
Each interaction with the service is a Command with execute() and undo() methods defined.
Maintain a stack of commands, push each command once it is successful.
When a command fails, i.e. interaction with a service fails, pop each command from the stack and call undo().
You might look for any Distributed Transaction APIs in .NET that would provide this functionality out of the box, though.
I would not recommend using distributed transactions between systems as it creates a lot of coupling between them and increases the load on each system as it holds locks for the others (you can see a write up I did on this called "transactional integration anti-pattern").
Usually a better option, as suggested by Vikdor is using a Saga which you can coordinate manually or by some external coordinator (e.g. using Orchestration).
Note that compensation logic might not be able to completely undo the operation (e.g. in a retail scenario, if an order caused you to hit an inventory threshold and order from your supplier a cancellation of that order doesn't have to cancel your order from your supplier. Also you may not refund the full sum etc. )
Related
I'm interested in using JMX to monitor/configure a simple Java Client/Server application. For example, we would capture any network exceptions that occur in a Java program.
Can MBeans be extended in this way? Or are they limited to more concrete get & set functions?
So far, I've looked in Notifications and Monitor MBeans
Thanks
Well I would say that it's definitely doable. I was using JMX in an Apache Wicket application earlier with custom MBeans. Anyway MBeans is just a wrapper around some logic in your server application. So you can take the data directly from your application.
If you want to take an example how is this done in a working application you might want to checkout this:
https://github.com/apache/wicket/blob/master/wicket-jmx/src/main/java/org/apache/wicket/jmx/wrapper/MarkupSettings.java
The class basically holds a reference to the application and asks for data directly form the server app.
When the server starts up, then it registers all the MBeans through an initializer class:
https://github.com/apache/wicket/blob/master/wicket-jmx/src/main/java/org/apache/wicket/jmx/Initializer.java
Then every time when you take a look in your MBean server you will see the latest up-to-date information coming directly from the app.
There are some caveats though. One caveat is that Java in general doesn't provide any good abstraction to capture all Exceptions of a given type coming from any source of the application. You can register your catch-all exception handler but as far as I can remember it doesn't work perfectly.
What I was doing when I had to do something like this, I was using AspectJ to register an all catch place to handle exceptions. I was using compile time weaving to reduce the performance implication but I am not sure how much does it affect the overall performance (if it affects at all).
¯\_(ツ)_/¯
The other caveat is that JMX connections are usually difficult to set up in an enterprise environment. If you have to log-in through two hops just to arrive to the production servers because there are firewalls everywhere than your monitoring connection will definitely fail and you need to keep buying beer to your sysadmin and convince your manager that this is not imposing any security risk. :)
There is one thing though. You say
to monitor/configure a simple Java Client/Server application
You want to configure / monitor the clients as well? I've never done that. I am not sure that's even possible.
This is a rather high-level question so apologies if it's off-topic. I'm new to the enterprise Java world.
Suppose I have written some individual Java packages that do things like parse data feeds and store the parsed information to a queue. Another package might read from that queue and ingest those entries into a rules engine package. Tripped alerts get fed into another queue, which is polled by an alerting service (assume it's written in Python) that reads from the queue and issues emails.
As it stands I have to manually run each jar file and stick it in the background. While I could probably daemonize some or all of these services for resiliency or write some kind of service manager to do the same, this strikes me as being very amateur. Especially since I'd have to start a dozen services for this single workflow at boot.
I feel like I'm missing something, but I don't know what I don't know. Short of writing one giant, monolithic application, what should I be looking into to help me manage all these discrete components and be able to (conceptually) deliver a holistic application? I'd like to end up with some sort of hypervisor where I can click one button, it starts/stops all the above services, provides me some visibility into their status and makes sure the services are running when they should.
Is this where frameworks come into play? I see a number of them but don't know if that's just overkill, especially if I'm not actively developing a solution for that framework.
It seems you architected a system with a lot of components, and then after some time you decided to aggregate some of them because they happen to share the same programming language: Java. So, first a warning: this is not the best way to wire components together.
Also, it seems you don't know Java very well because you mix terms like package, jar and executable that are totally unrelated and distinct concepts.
However, let's assume that the current state of the art is the best possible and is immutable. Your current requirement is building a graphical interface (I guess HTTP/HTML based) to manage all the distinct components of the system written in Java. I suggest you use a single JVM, writing your components as EJB (essentially a start(), stop() and a method to query the component state that returns a custom object), and finally wire everything up with the Spring framework, that has a nice annotation-driven configuration for #Bean's.
SpringBoot also has an actuator package that simplify exposing objects. You may also find it useful to register your beans as Managed beans, and using the Hawtio framework to administer them (via a Jolokia agent).
I am not sure if you're actually using J2EE (i.e. Java Enterprise Edition). It is possible to write enterprise software also in J2SE. J2SE is not having too much available off the shelf for this, but in contrast has a lot of micro-frameworks such as Ninja, or full stack frameworks such as Play framework which work quite well, much easier to program, and performs much better than J2EE.
If you're not using J2EE, then you can go as simple as:
make one new Java project
add all the jars as dependency to that project (see the comment on Maven above by NimChimpsky)
start the classes in the jars by simply calling their constructor
This is quite a naive approach, but can serve you at this point. Of course, if you're aiming for a scalable platform, there is a lot more you need to learn first. For scalability, I suggest the Play! framework as a good start. Alternatively you can use Vert.x which has its own message queue implementation as well as support for high performance distributed caches.
The standard J2EE approach is doable (and considered "de-facto" in many oldschool enterprises) but has fundamental -flaws- or "differences" which makes a very steep learning curve and a very much non-scalable application.
It seems like you're writing your application in a microservice architecture.
You need an orchestrator.
If you are running everything in a single machine, a simple orchestrator that you probably is already running is systemd. You write systemd service description, and systemd will maintain your services according to your services description. You can specify the order the services should be brought up based on dependencies between services, restart policy if your service goes down unexpectedly, logging for stdout/stderr, etc. Note that this is the same systemd that runs the startup sequence of most modern Linux distros.
If you're running multiple machines, you can still keep using single machine orchestrator like systemd, but usually the requirement for the orchestrator will also become more complex. With multiple machines, you now have to take into account things like moving services between machines, phased roll out, etc. For these setups, there are software that adapts systemd for multi machine orchestration, like CoreOS's fleetd; and there are also standalone multi machine orchestrator like Kubernetes. Both uses docker as application container mechanism.
None of what I've described here is Java specific, which means you can use the same orchestration for Java as you used for Python or other languages or architecture.
You have to choose, As Raffaele suggested you can choose to write all your requirements into one app/service. Seems like a possible mission, using java Ejb's or using spring integration - ampqTemplate ( can write to a queue with ampqTemplate and receive the message with a dedicated listener (example).
Or choosing implementation with microservices architecture. write a service that will push to the queue another one that will contain the listener etc. a task that can be done easily with spring boot.
"One button to control them all" - in the case of a monolithic app - it's easy.
In case that you choose microservices architecture. It depends what are you needs. if its just the "start" "stop" operation I guess that that start and stop of your tomcat/other server will do. For other metrics, there is a variety of solutions. again, it depends on your needs.
I made a web based application by using the java language, and I would like to monitor its performance periodically (e.g. response time). Also I want to display this information on the homepage of my application. Is that possible? Can I have any idea about how this can be made.
Thanks.
You can take a look at stagemonitor. It is a open source java web application performance monitor. It captures response time metrics, JVM metrics, request details (including a call stack captured by the request profiler) and more. The overhead is very low.
Optionally, you can use the great timeseries database graphite with it to store a long history of datapoints that you can look at with fancy dashboards.
Example:
Take a look at the github page to see screenshots, feature descriptions and documentation.
Note: I am the developer of stagemonitor
Depending on your environment, I would use a cron job or task that measures the response time to request your app using something like HttpClient. Then drop that information into a database table accessible by your app.
The answer here is the simplest way you can measure the time: How do I time a method's execution in Java?
Why not checkout Munin monitoring? The website says
Munin the monitoring tool surveys all your computers and remembers
what it saw. It presents all the information in graphs through a web
interface. Its emphasis is on plug and play capabilities. After
completing a installation a high number of monitoring plugins will be
playing with no more effort.
SLAC at the Stanford university also keeps a large, quite well sorted list with various solutions for network monitoring among other things. SLACs list of Network Monitoring Tools, check for instance "Public domain or free network monitoring tools".
You can also consider to create your own custom web application monitor. Therfore, use the ProxyPattern and and create a concreate monitor. By using Spring framework you can easily swich on and off the monitor during runtime without re- deployment or restart of the web application. Furthermore you can create a lot of different specific monitors by yourself and are able to control what is beeing monitored. This gives you a maximum of flexibility, but requires a bit of work.
It is possible.
The clearest way to go about it, providing true numbers is to simulate a client that performs some sort of activity that mimics the real usage. Then have that client periodically use the website.
This presupposes that your website has a means to accept inputs that do not impact the real back end business. Crafting such interfaces requires some thought, but is not beyond the ability of a person who could put together the web site in the first place. The key points are to attempt to emulate as much using the real website as possible, but guard against real business impact. Basically it is designing for a special user (the tester).
So you might have a special user that when logged in, all purchases are bound to a special account that actually is filtered out to appropriately not demand payment and not ship goods. Provided the systems you integrate with all share an understanding of this live testing account, you can simultaneously test alongside of real production post-deployment.
Such a structure provides a huge benefit. You get performance of the real, live running system. Performance tends to change over time, and is subject to the environment. By fetching your performance numbers on the live system, in the same environment, you get a much better view of what real users might be encountering. Also, you can differentiate and track performance for different activities.
Yes, it is a lot more to design and set up; however, if you are in it for the long run, the benefits are huge.
I guess JavaMelody is the most appropriate solution for you. It can be built into a Java application and due to this feature, it monitors the functionality inside the app. Using this platform, it’s possible to get much more specific parameters for your Java app, than via external monitoring. In addition, it allows you displaying some statistics on your app homepage. Moreover, you can build in the app the graphs from JavaMelody, that essentially facilitates the app monitoring.
Take a look at the detailed overview of JavaMelody: http://cases.azoft.com/enterprise-system-monitoring-solutions-business-apps/
I am writing a small proxy application which should be redundant, e.g. primary proxy will be running on one server and the redundant one will run on a separate server. Is there a simple high-availability framework which I can use to implement this redundancy? For example, this HA framework would send pings between instances and raise some sort of exception or notification on the other instance when the first one goes down.
Building such a system has been my routine job in recent years. I have found jgroups
a very usable tools to receive and handle such kind of grouping events. This is the case if you want to build your own HA infrastructure. I don't know, but maybe in your case just a simple reverse proxy such as HAProxy can be enough.
If you want HA without hassle, just use some load balancer with HA capability e.g. Ultramonkey, LVS with keepalived etc.
In a HA configuration, you'd typically want to use virtual IP, so even if you'd have this ping/notify functionality as a framework, you'll still have stuff to do (start responding to requests to the virtual IP once the other instance has failed). So unless you are looking for a learning occasion, I'd advice using a middleware instead of coding this yourself using frameworks.
There are number of health-checks that you can configure for these middlewares. A simple healthcheck might for example, fire a GET request to your app. periodically and look for a specific string (e.g. "XXX running.") in the response to make sure your app. is running fine.
You don't provide much details about the work your application does, so depending on how stateful it is, whether it can tolerate minor dataloss, is it time-critical, do you value developer time over machine time, you can have a varying spectrum of solutions.
There are some good suggestions above, I'd add: take a look at JMS and persistent messaging. Usually these make recovery quite trivial, but at the cost of latency hit (unless you byu a commercial product and learn it well or pay the vendor to tune your application). With JMS queues you can implement active-active processing and save yourself the headache of failure detection.
Another direction to look at is distributed state management/clustering framework like Gigaspaces, Coherence, Gemstone, Infinispan, Gridgain and Teracotta. These can replicate your data and guarantee varying quality of services levels. Most of them come with some type of failure detection and distributed management mechanism.
hadoop is a good place to start
I'm looking for a straightforward Java workflow engine that:
can handle both automated and manual (GUI-based) steps within a workflow
supports long-running, asynchronous tasks
provides support for restarting workflows in the event of a server crash
stores a full audit history of previously executed workflows
provides easy access to this audit history data
Possible candidates include the new Drools Flow process engine in Drools 5, and OSWorkflow from OpenSymphony. From my current understanding, OSWorkflow seems to offer more of what I want (Drools Flow doesn't appear to store much in the way of an audit history); however, the most recent release of OSWorkflow was back in early 2006. Is it a mistake to now use OSWorkflow when it's no longer under active development?
Does anyone have much experience with either/both of these frameworks? Are there any other workflow engines I should be looking at? All recommendations welcome - thanks.
Just to clarify how Drools Flow supports the requirements you are describing (refering to the Drools Flow documentation):
can handle both automated and manual (GUI-based) steps within a workflow
Drools Flow uses (domain-specific) work items (Chapter 8) to interact with external systems. These could be automated services, or a human task management component (Chapter 9) for manual tasks. This human task component is fully pluggable but Drools Flow supports a WS-HumanTask implementation out of the box. Drools 5.1 will include web-based task lists, including custom task forms.
supports long-running, asynchronous tasks
The engine allows you to start processes that can live for a long time. The process supports different kinds of wait states (work item nodes, event nodes, event wait nodes, sub-process, etc.) to model long-running processes. External tasks can be integrated synchronously or asynchronously.
provides support for restarting workflows in the event of a server crash
The runtime state of all process instances can easily be stored in a data source by turning on persistence (Chapter 5.1). Therefore, all processes can simply be restored in the state they were in after a server crash.
stores a full audit history of previously executed workflows
Drools Flow generates events about what is happening during the execution of your processes. By turning on audit logging (Chapter 5.3), these events can be stored in a database, providing a full audit history of whatever happened during execution.
provides easy access to this audit history data
The history data is stored using a few simple database tables. These tables can be queried directly, or you could use it for generating custom reports (Chapter 12.1) that show the key performance indicators that are relevant for your application.
Furthermore, we believe that a knowledge-oriented approach, allowing you to seamlessly combine processes with rules and event processing whenever necessary, will offer you more power and flexibility compared to aforementioned process-oriented engines.
Kris Verlaenen
Drools Flow Lead
I've not had any experience with the candidates you've mentioned but judging from the projects that I've worked on it might be worth looking at jBPM. Quite a few developers I've worked with swear by it and I think it fits your criteria quite nicely.
Drools Flow is at lot more sophisticated and powerful than both jBPM and OSWorkflow and development is moving at a faster pace than either. We provide a lot of details and screenshots here:
http://www.jboss.org/drools/drools-flow.html
But in summary. You get interactive debugging across rules, workflow and event processing. You have a larger set of built in nodes, improving the number of problems you can directly model declaratively. Correlated (across rules, processes and events) audit logging and reporting. We provide a very simple and yet powerful mechanism for building domain specific workflow, via our pluggable work items.
Drools 5.0 has just been released and 5.1 is going to follow in the next 4 to 6 weeks. We are adding simulation and testing for this, using an MVEL DSL, which we believe will be a huge hit. This will also include more extensive work for remote admin GUIs for processes, all integrated into Guvnor.
The Drools team also prides itself on being more accessible than any of those other mentioned projects. Feel free to pop onto irc for a chat.
Mark
I have experience with both.. also I was involved in a tool for migrating existing processes in OSWorkflow to Drools 5.0. You can read an article about that in: http://blog.athico.com/2009/01/drools-flow-and-osworkflow-migration.html. It is important to mention that this migration tool/translator was created to allow old projects that are using OSWorkflow to upgrade to Drools 5.0 and take advantages of all the Drools 5.0 Business Integration Platform.
Greetings