We are developing a vehicle tracking system. Like every VTS, we have GPS devices fitted into the vehicles which keep sending location information to the server. On server, our TCP communicator process keeps reading that data and saves it into the database
Now, we need to check for some set of rule to trigger alerts for the vehicles, e.g We need alert when vehicle reaches to a particular location, if vehicle crosses specific speed-limit,etc.
Can you please suggest the best way to implement it?
We have thought of some ways to implement it,
1. Our TCP communicator, when receives the location, should check for the alerts.
2. There will be a process which will keep running every 15 minutes and check the location details in that 15 minutes for alerts.
I am looking for the suggestions to implement it, logic-wise as well as technology-wise. e.g. Whether we should use Drools or not?, etc.
Somebody from FedEx actually presented something like this in a JavaOne conference I attended a couple of years back.
Basically, the idea was, yes, using Drools Expert + Fusion to perform CEP (complex events processing) on vehicle location data.
As far as I can recall, a vehicle would periodically (every couple of seconds even) send its GPS coordinates to the engine (an event) which would then be digested by the rules engine, and depending on the rules could trigger certain actions such as raising alerts ("vehicle is stalled" or "out of course") or sending notifications ("vehicle will arrive at destination in ~15 minutes").
(Google for "drools fusion cep vehicle tracking" uncovers this presentation which should give you few more details or at least provide some insight.)
the way Drools work, is that you fill in alot of objects into the "Working Memory" of Drools. While you fill in the objects, Drools will find out which rules "fires" on the objects and stores the objects in a Rete-Tree. When you are finished putting the objects in the memory and you fire all rules, Drools will process the code you wrote corresponding to the rule.
I would suggest, that you make an object with all the data recieved from the vehicle necessary for your rules and put it in the working memory.
In Drools you should make many small rules, each just checking one thing and acting on the result.
It is not a good practice to let Drools get data needed for evaluation, but I can't see any problems in letting Drools trigger some events, that send messages to a vehicle or some other system. (I guess that should happen async, so that you don't slow down Drools) In fact, Drools offers you to hook up an eventlistener.
There's no reason to run every 15 minutes. That will introduce delay in the triggers and also result in bursts of load every 15 minutes followed be periods of no load.
You can have a flag in your database for new alert rules and new location data. When you scan for events, you can use a two-pass approach. Check all new rules against all location data and mark them no longer new. Then check all new location data against existing rules and mark them no longer new.
You can run this as often as you like. Ideally, you wouldn't wait that long because the longer you wait, the more work you accumulate.
As for having the TCP communicator check for relevant alerts over the scan the database periodically approach, the main advantage would be alerts would be immediate. The disadvantage would be that alert processing would slow down the TCP communicator path and you would be locked into a "one update means one check for alerts" model.
In the "scan the database" approach, if load gets too high, you can wind up checking for alerts only on every so many updates from high-frequency update sources. This naturally deals with load by reducing the amount of work needed, but it could result in a missed alert.
I think all the approaches you're considering will work fine.
Related
Let's say I have an API that gives me the values of stock for the last month. The data is sampled every hour.
Now I want to make a web app that would visualize this data on a line chart. I don't need all the hourly samples, so my question is how should I make this work?
My idea is that there would be a backend app (i.e. in Java Spring) that would GET the data from the API and calculate the average for each day (using a stream, maybe parallel stream?) and then put that in a new collection and pass it on to the front end to put in a chart.
Start thinking from the UI, what do you need there, how often do you need it and how fast ?
Then get the data from the backend, if there is too much data at once and the API cannot do otherwise, either:
get data and reduce to what the UI needs (backend), use once and throw away
OR get data and reduce to what the UI needs (backend), keep in cache for a while
OR pre-process the data so that when the UI needs it, it will be ready
For the return format, consider something lightweight, like some simple named json array {"dayAverages": [0.34, 1253.432, ...]}, "month" : 2, "year": 2018}, then in the UI adapt to the needs of your lib (that is debatable).
Also observe how users use the UI, then you may get some ideas on how to optimize the experience (preload next month ...)
If you do this for learning purposes, consider doing it async + lambdas = bonus :)
As to your question "...how should I make this work?" --
This is extremely broad. There are many, many ways to do this. Some of these ways depend heavily on your architecture, how much traffic is expected to your app, what request-load the API can handle, etc. Here are a few general things to consider:
Any sort of MVC architecture (or similar) would be a good fit for your Web app.
You mention needing a "backend app" of some type. Not sure what you mean here, but the averaging features can be built directly into your Web app framework without needing a separate back-end app.
If you're going to calculate averages for display in the Web app, you will need to maintain state somewhere. Assuming the API doesn't give this to you, you'll need a database of some type, or at least some type of memory caching storage engine to facilitate this. How you do this will depend on your architecture and the traffic/load on your app (e.g. will you have multiple, load-balanced servers).
Hope that helps. We could give more if you ask some specific questions.
I read the "The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in MassiveScale, Unbounded, Out of Order Data Processing" paper. Alas, the SDK does not yet expose the accumulating & retracting triggering mode (section 2.3).
I was wondering if there was a workaround for getting similar semantics?
I have been reading the source and have figured out that StateTag or StateNamespace may be the way i can store the "last emitted value of the window" and hence can be used to calculate the retraction message down the pipeline. Is this the correct path or are there other classes/ways I can/should look at.
The upcoming state API is indeed your best bet for emulating retractions. Those classes you mentioned are part of the state API, but everything in the com.google.cloud.dataflow.sdk.util is for internal use only; we technically make no guarantees that the APIs won't change drastically, or even remain unreleased. That said, releasing that API is on our roadmap, and I'm hopeful we'll get it released relatively soon.
One thing to keep in mind: all the code downstream of your custom retractions will need to be able to differentiate them from normal records. This is something we'll do automatically for you once bonafide retraction support is ready, but in the mean time, you'll just need to make sure all the code you write that might receive a retraction knows how to recognize and handle it as such.
I am creating an app that checks for user locations every half an hour and updates the location of the user in the local database and then runs CRUD queries based on the user's location even when the app is not running. How do i do it ?
I have referred to this http://techtej.blogspot.com.es/2011/03/android-thread-constructspart-4.html article and i am still confused about which is the correct approach for my result ?
There are 4 options according to the article for what i intend to achieve according to me
1) Service : But since i feel it would be a long operation with the local database, i feel i should ignore this one.
2) IntentService : This cannot perform multiple tasks, so i feel this one also should be avoided for me as i have to get the location of the user and scan the database , update the database (3 tasks)
3)Thread : I am not sure how to call this when the app is not open
4) AsyncTask : I am not sure how to call this when the app is not open.
Basically i looking for something like a CRON JOB that runs on a local database while working on the location data.
It would be great if you could link me up to some tutorials and answer with a simple example to make me understand the difference of all 4 methods.
// editted on 16 March :
I have read something about a JobScheduler which is introduced in the API 21, but not sure if it also supports till Gingerbread and is it the right approach for my question
Thanx
When recording the users position use a service with a notification. Just for the sake of creating a morally responsible app that informs the user the app is tracking them. The service by definition runs in the background.
A fused location provider with setinterval(long) 30 minutes gets the interval. Set fastestInterval() to a minute to receive GPS data when other apps are using the GPS.
Have you considered using a SyncAdapter. Its best to schedule jobs at fixed interval and also optimized for battery usage. Also, once started, it can run independently of the app. As per your requirements, I believe this is best suited for your need. You can read about this here. This also removes the corner case of starting the service (generally used) when your device is restarted. Your app will still continue running the scheduled job even if the device gets restarted.
In the SyncAdapter you have to use a ContentProvider so wrap your DB inside a ContentProvider. Also, preferably use a CursorLoader to run longrunning tasks on DB. You should read about CursorLoader. This is a great way to access your resources. Also, you can define an Observer Design Pattern which Observes for changes in a DB and will perform a task when changes are made in DB. This can also be used inside your application itself and also inside SyncAdapter. Cursor Loader is best preferred for background work on DB. You can perform all CRUD Operations using a CursorLoader and ContentProvider.
This cannot perform multiple tasks
Yes, it can. It has only one thread, and so it can only do one simultaneous task.
i have to get the location of the user and scan the database , update the database (3 tasks)
I have no idea why you think that is three tasks. You cannot do them simultaneously.
Your bigger problem with IntentService is that getting location data is asynchronous, and IntentService is not well-suited for calling APIs that themselves are asynchronous.
But since i feel it would be a long operation with the local database, i feel i should ignore this one.
The point behind any service is for "a long operation".
Basically i looking for something like a CRON JOB that runs on a local database while working on the location data
Use AlarmManager to trigger a WakefulBroadcastReceiver, which then triggers a Service. The Service, in onStartCommand(), forks a background thread to (asynchronously) retrieve the location and update the database. The Service can then call completeWakefulIntent() on WakefulBroadcastReceiver, plus stopSelf() with the startId received in onStartCommand() for this work, plus allow the thread to terminate. If no other commands were received in the interim, the service will shut down.
I think you are looking for something similar to WakefulIntentService. This handles all your cases completely.
You can do your location and db related work inside doWakefulWork() of said implementation.
I've done what you are looking for, both with GPS and non-GPS.
The project I took as staring point for the non-GPS solution already does all you need, and is battery-friendly (credits should go to Kenton Price):
https://code.google.com/p/little-fluffy-location-library/
Take a look at it, it works like a charm. Just run it in any device. If you need any help customizing just let me know.
Just edit the "onReceive" method in the "TestBroadcastReceiver" to update your DB.
If you need the GPS solution let me know too, but I dropped it for being a battery killer!
Hope it helps.
1. I think for this requirement, Thread and inside it AsyncTask -- this structure will be useful.
In link provided by you, it is mentioned very nicely here
2. For location related blog, you can check useful materials here :
(1) Difference between Google Map Distance and Directions API
(2) Check this answer also
Hope this will help you
I'm writing a system that will leverage Mongo for persistence and RabbitMQ for a message bus/event queueing, and I'm trying to figure out the best way to be resilient to failures on the publication side.
There are three scenarios I can think of:
Everything works - consistent
Everything fails - consistent
Part works, whichever happens later is out of date - inconsistent
The last case is the one I'm interested in, and I'm curious to know how others have solved the issue, given that XA isn't an option (and I wouldn't want the performance overhead anyway).
There are a couple of solutions I can think of:
Add a "lastEvent" (or some similar) field to the Mongo document. On a periodic interval, scan for documents where lastEvent < lastUpdated, and fire an event (this requires an extra update for every change, and loses context of the "old" document in the case of an update)
Fire the event in Rabbit before persisting in Mongo, and allow safe handling of events that may not have actually happened (really dislike this approach)
Could anyone else shed some light on how to provide some sort of consistency across a persistence layer and message bus?
1 is never a good idea, the notion of "last X time" falls over as soon as you introduce multi-threaded or multi-process systems, and when that "time" is generated (if some requests take longer to process then others, then the "later" time might be written before the "earlier" times to the persistent store)
2 Is basically Idempotence, and it's a pattern that works very well for designing fault tolerant systems if done properly
I have a lot of existing data in my database already, and want to develop a points mechanism that computes a score for each user based on what actions they do.
I am implementing this functionality in a pluggable way, so that it is independent of the main logic, and relies on Spring events being sent around, once an entity gets modified.
The problem is what to do with the existing data. I do not want to start collecting points from now, but rather include all the data until now.
What is the most practical way to do this? Should I design my plugins in such a way as to provide for an index() method, which will force my system to fetch every single entity from the database, send an EntityDirtyEvent, to fire the points plugins, for each one, and then update it, to let points get saved next to each entity. That could result in a lot of overhead, right?
The simplest thing would be to create a complex stored procedure, and then make the index() call that stored procedure. That however, seems to me like a bad thing either. Since I will have to write the logic for computing the points in java anyway, why have it once again in SQL? Also, in general I am not a fan of splitting business logic into the different layers.
Has anyone done this before? Please help.
First let's distinguish between the implementation strategy and business rules.
Since you already have the data, consider obtaining results directly from the data. This forms the data domain model. Design the data model to store all your data. Then, create a set of queries, views and stored procedures to access and update the data.
Once you have those views, use a data access library such as Spring JDBC Template to fetch this data and represent them into java objects (lists, maps, persons, point-tables etc).
What you have completed thus far does not change much, irrespective of what happens in the upper layers of the system. This is called Model.
Then, develop a rule base or logic implementation which determines, under what inputs, user actions, data conditions or for all other conditions, what data is needed. In mathetical sense, this is like a matrix. In programming sense, this would be a set of logic statements. If this and this and this is true, then get this data, else get that data, etc. This encompasses the logic in your system. Hence it is called "Controller".
Do not move this logic into the queries/stored procedure/views.
Then finally develop a front-end or "console" for this. In the simplest case, develop a console input system, which takes a .. and displays a set of results. This is your "view" of the system.
You can eventually develop the view into a web application. The above command-line view can still be viable in the form of a Restful API server.
I think there is one problem here to be considered: as I understand there's huge data in the Database so the idea to create only one mechanism to calculate the point system could not be the best approach.
In fact if you don't want to start collecting points but include all the data, you must process and calculate the information you have now. Yes, the first time you will run this can result an overhead, but as you said, you need this data calculated.
By other hand you may include another mechanism that attends changes in an entity and launches a different process capable of calculate the new pointing diffence that applies to this particular modification.
So, you can use one Service responsible of calculate the pointing system, one for a single entity and another, may be longer to finish, capable of calculate the global points. Even, if you don't need to be calculated in real-time you can create a scheduled job responsible of launch it.
Finally, I know it's not a good approach to split the business logic in two layers (Db + Java) but sometimes is a requirement do it, for example, if you need to reply quickly to a request that finally works with a lot of registries. I've found some cases that there's no other option than add business logic to the database (as a stored procedures, etc) to manage a lot of data and return the final result to the browser client (ex: calculation process in one specific time).
You seem to be heading in the right direction. You know you want your "points" thing decoupled from the main application. Since it is implied you are already using hibernate (by the tag!), you can tap into the hibernate event system (see here section 14.2). Depending upon the size/complexity of your system, you can plugin your points calculations here (if it is not a large/complex system), or you can publish your own event to be picked up by whatever software is listening.
The point in either design approach is that neither knows or cares about your point calculations. If you are, as I am guessing, trying to create a fairly general purpose plugin mechanism, then you publish your own events to that system from this tie-in point. Then if you have no plug-ins on a given install/setup, then no one gets/processes the events. If you have multiple plug-ins on another install/setup, then they each can decide what processing they need to do based upon the event received. In the case of the "points plugin" it would calculate it's point value and store it. No stored proc required....
You're trying to accomplish "bootstrapping." The approach you choose should depend on how complicated the point calculations are. If stored procedures or plain update statements are the simplest solution, do that.
If the calculations are complicated, write a batch job that loads your existing data, probably orders it oldest first, and fires the events corresponding to that data as if they've just happened. The code which deals with an event should be exactly the same code that will deal with a future event, so you won't have to write any additional code other than the batch jobs themselves.
Since you're only going to run this thing once, go with the simplest solution, even if it is quick and dirty.
There are two different ways.
One is you already know that - poll the database for for changed data. In that case you are hitting the database when there may not be change and it may slow down your process.
Second approach - Whenever change happens in database, the database will fire the event. That you can to using CDC (Change Data Capture). It will minimize the overhead.
You can look for more options in Spring Integration