I'm writing a system that will leverage Mongo for persistence and RabbitMQ for a message bus/event queueing, and I'm trying to figure out the best way to be resilient to failures on the publication side.
There are three scenarios I can think of:
Everything works - consistent
Everything fails - consistent
Part works, whichever happens later is out of date - inconsistent
The last case is the one I'm interested in, and I'm curious to know how others have solved the issue, given that XA isn't an option (and I wouldn't want the performance overhead anyway).
There are a couple of solutions I can think of:
Add a "lastEvent" (or some similar) field to the Mongo document. On a periodic interval, scan for documents where lastEvent < lastUpdated, and fire an event (this requires an extra update for every change, and loses context of the "old" document in the case of an update)
Fire the event in Rabbit before persisting in Mongo, and allow safe handling of events that may not have actually happened (really dislike this approach)
Could anyone else shed some light on how to provide some sort of consistency across a persistence layer and message bus?
1 is never a good idea, the notion of "last X time" falls over as soon as you introduce multi-threaded or multi-process systems, and when that "time" is generated (if some requests take longer to process then others, then the "later" time might be written before the "earlier" times to the persistent store)
2 Is basically Idempotence, and it's a pattern that works very well for designing fault tolerant systems if done properly
Related
I'm currently struggling to wrap my head around some concurrency concepts in general. Suppose we have a REST api with several endpoints for updating and creating entities in our database. Lets assume that we receive 100 concurrent requests at the same time for a certain update. How do we guarantee that our data consistency is retained? If working with Java, I guess some options would be:
Using lock mechanisms
Using synchronization on methods in our service layer
However surely this would make a huge impact on the scalability of our application? But I can't see any other way currently of ensuring that we don't encounter any race conditions when interacting with our database. (Also I dont think there is much point to adding synchronization to every method we write in our service?)
So, I guess the question is: how can we reliably secure our application from race conditions with concurrent requests, while at the same time retaining scalability?
I realize this is a very open-ended and conceptual question, but please if you can point me in the right direction of what area / topic I should dive into for learning, I would be grateful.
Thanks!
You got a good understanding of the problem.
You have to decide between eventual consistency and strong consistency. Strong consistency will limit scaling to a certain extent but you also really need to sit down and be realistic/honest about your scaling needs(or your consistency needs).
It's also possible to limit consistency for example rows in a database could be consistent or you can be consistent geographically within a region or a continent. Different queries can also have different requirements.
Creating efficient and strongly consistent databases is a whole field of research and all the big tech giants have people working on that, there are too many solutions/technologies to list. Just googling something like "strong consistency scaling" will get you a ton of results you can read.
We are running a setup locally where we start two instances of an Axon application. The following properties are set in application.yml:
axon:
eventhandling:
processors:
SomeProcessorName:
initialSegmentCount: 2
threadCount: 1
mode: TRACKING
So both nodes have a single thread and they should each process a segment. They both connect to AxonServer. How do the two instances coordinate segment claims?
If I start both of these applications using an in-memory database, I can see in AxonServer that they both attempt to claim segment 0 and that segment 1 is claimed by neither. (We get a duplicated claim/unclaimed segment warning). If they connect to the same database, this does not happen, instance 1 claims segment 0, instance 2 claims segment 1.
Am I then correct in assuming that identical processors have to share a database in order for this to work properly? I can't find this information immediatly in the reference docs.
Does this then also mean that if I would hypothetically want to replicate a projection model for performance reasons (e.g: database server in the US and another one in the EU), this would not work properly?
To clarify: I would want both databases to build an identical query model that could both be queried separately. As it is right now (assuming that we could run two nodes on two databases), node 1 would only process events for segment 0, node 2 would only process events for segment 1. If I understand this correctly, this means that both databases only contain half of the information of the query model.
So in order to pull this off, I would have to create another near-identical codebase, with the only difference being the processor name?
I think I can give some guidance in this area.
Axon Server does not provide coordination between Tracking Tokens of TrackingEventProcessor at this point in time.
Thus, coordination of this part is purely in your application environment, or differently put, with the Axon Server client.
The most pragmatic approach would be to share the underlying storage solution for your TokenStore between both application; so your assumption on this part is correct.
Current implementations of the TokenStore are indeed database-based - nothing stops you to come up with a distributed solution of this though, as this is all open source and freely adjustable.
I do not completely follow your hypothetical suggestion that:
Does this then also mean that if I would hypothetically want to replicate a projection model for performance reasons (e.g: database server in the US and another one in the EU), this would not work properly?
Well, this can work properly, but I think the segmentation of a given TrackingEventProcessor it's TrackingToken is not the way to go in this part.
This solution is intended to share the work load of updating a single Query Model.
The 'work load' in this scenario is the Event Stream by the way.
If you're looking to replicate a given Query Model by means of reading the Event Stream, I'd indeed suggest to have a second TrackingEventProcessor, which has an identical Event Handling Component underneath.
Note that this should not require you to 'replicate the code base'.
You should merely need to register two Event Handling Components to two distinct TrackingEventProcessors.
If you are using Spring Boot as configuration, all this is typically abstracted away from you. But if you take a look at the EventProcessingConfigurer, you should be able to find a fair API describing how to achieve this. If things aren't clear in that area, I'd suggest a different issue should be introduced, as the topic somewhat diverges from the original question.
Hoping this is sufficient for you to proceed #MatthiasVanEeghem!
I read the "The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in MassiveScale, Unbounded, Out of Order Data Processing" paper. Alas, the SDK does not yet expose the accumulating & retracting triggering mode (section 2.3).
I was wondering if there was a workaround for getting similar semantics?
I have been reading the source and have figured out that StateTag or StateNamespace may be the way i can store the "last emitted value of the window" and hence can be used to calculate the retraction message down the pipeline. Is this the correct path or are there other classes/ways I can/should look at.
The upcoming state API is indeed your best bet for emulating retractions. Those classes you mentioned are part of the state API, but everything in the com.google.cloud.dataflow.sdk.util is for internal use only; we technically make no guarantees that the APIs won't change drastically, or even remain unreleased. That said, releasing that API is on our roadmap, and I'm hopeful we'll get it released relatively soon.
One thing to keep in mind: all the code downstream of your custom retractions will need to be able to differentiate them from normal records. This is something we'll do automatically for you once bonafide retraction support is ready, but in the mean time, you'll just need to make sure all the code you write that might receive a retraction knows how to recognize and handle it as such.
This is one of the questions that involves crossing what I call the "Hello World Gulf" I'm on the "Hello world" I can use SQLite and Content Providers (and resolvers) but I now need to cross to the other side, I cannot make the assumption that onUpgrade will be quick.
Now my go-to book (Wrox, Professional Android 4 development - I didn't chose it because of professional, I chose it because Wrox are like the O'Reilly of guides - O'Reilly suck at guides, they are reference book) only touches briefly on using Loaders, so I've done some searching, some more reading and so forth.
I've basically concluded a Loader is little more than a wrapper, it just does things on a different thread, and gives you a callback (on that worker thread) to process things in, it gives you 3 steps, initiating the query, using the results of the query, and resetting the query.
This seems like quite a thin wrapper, so question 1:
Why would I want to use Loaders?
I sense I may be missing something you see, most "utilities" like this with Android are really useful if you go with the grain so to speak, and as I said Loaders seem like a pretty thin wrapper, and they force me to have callback names which could become tedious of there are multiple queries going on
http://developer.android.com/reference/android/content/Loader.html
Reading that points out that "they ought to monitor the data and act upon changes" - this sounds great but it isn't obvious how that is actually done (I am thinking about database tables though)
Presentation
How should this alter the look of my application? Should I put a loading spinning thing (I'm not sure on the name, never needed them before) after a certain amount of time post activity creation? So the fragment is blank, but if X time elapses without the loader reporting back, I show a spiny thing?
Other operations
Loaders are clearly useless for updates and such, their name alone tells one this much, so any nasty updates and such would have to be wrapped by my own system for shunting work to a worker thread. This further leads me to wonder why would I want loaders?
What I think my answer is
Some sort of wrapper (at some level, content provider or otherwise) to do stuff on a worker thread will mean that the upgrade takes place on that thread, this solves the problem because ... well that's not on the main thread.
If I do write my own I can then (if I want to) ensure queries happen in a certain order, use my own data-structures (rather than Bundles) it seems that I have better control.
What I am really looking for
Discussion, I find when one knows why things are the way they are that one makes less mistakes and just generally has more confidence, I am sure there's a reason Loaders exist, and there will be some pattern that all of Android lends itself towards, I want to know why this is.
Example:
Adapters (for ListViews) it's not immediately obvious how one keeps track of rows (insert) why one must specify a default style (and why ArrayAdapter uses toString) when most of the time (in my experience, dare I say) it is subclasses, reading the source code gives one an understanding of what the Adapter must actually do, then I challenge myself "Can I think of a (better) system that meets these requirements", usually (and hopefully) my answer to that converges on how it's actually done.
Thus the "Hello World Gulf" is crossed.
I look forward to reading answers and any linked text-walls on the matter.
you shouldnt use Loaders directly, but rather LoaderManager
Let say that there are events that may occur (are less likely), but they should be registered. This is data that one just need for tuning ... to see what has happened and what needs to be changed and improved.
Typically this is done in catch blocks when exception is thrown or just if some if condition passes.
I don't want to write shell scripts and collect data from logs so I can either use DB and create a table for every context, which is possible in most cases, but it is extremely inconvenient for maintenance and refactoring in further development. Especially because the data will by typed as RDB is used. Mostly the only shared data is userId, time, component and varying data like fileId, fileSize || elementsCount, count deviance || etc.
Or I can use some nosql store for that, which is also a little overkill to do that just because of this, but as the data has rather type free nature, I guess it would be more convenient.
Could you please explain how you do it ? How is this even called ? I think that JMX doesn't deal with this scenario. Spring AOP or AOP in general deals only with the distributed nature of this.
It sounds like you have two separate questions:
How should I capture certain events in the first place?
What should I do with the events once I've captured them?
Regarding 1, yes, AOP is a pretty common solution for capturing events. I'm not sure what you mean by "AOP in general deals only with the distributed nature of this". There's nothing distributed about AOP. You haven't told us enough about your application for anybody to say how to integrate AOP as easily as possible, etc, but lots of information is available online.
Regarding 2, how much data are we talking about? How much information do you want to store per event? What's similar/different about each message? I'd probably take the following approach:
Figure out how much data you're going to save during any given second, minute, hour, day, etc. If it's small enough to fit into one of your existing databases, then don't complicate your environment by introducing a new technology.
Can the data be loaded synchronously? If yes, then that's easy. Otherwise, I probably would log the data, and consolidate it periodically with a simple ETL script. This probably will be a whole lot easier and cheaper than setting up a new nosql store that you don't have in production now.
Decide what data you want to keep. It probably will look something like: id, type, timestamp, source (e.g. server or instance of the application), details. Details should be type-specific.
Decide what types of queries or reports you want to run on the data.
Do you need to structure the type-specific stuff so that specific queries are possible? Can you keep the type-specific stuff in an XML or JSON document, and only parse them in type-specific reports? Or, do you need to refer to type-specific stuff in the queries themselves? Type-specific details can make queries hard, but a nosql database such as mongodb might actually help here.
Figure out your data retention policy. You probably need to clean up old data at some point. This might affect the design of your storage.