I have 4 different raspberry pi's running the same program on each, the program sends information to a mySQL DB to be inserted into a table.
It is possible for this to happen or what problems will occur?
e.g.
Rpi:1 accessed -> sends info to DB
Rpi:2 accessed -> sends info to DB
Rpi:3 accessed -> sends info to DB
can these happen simultaneously?
I dont have 4 devices at the minute thats why i haven't tried it but i'm just wondering how this would work or if it is possible.
Revised : Cheers for the responses guys, each of the RPi are connected to a RFID module so when the fob get read, it send the timestamp to a DB and thats the same with all the 4 devices! Each device will be used at a random time when some one wants to access the system, will this cause problems?
Thanks :)
As #mastah indicates it really depends on what you want to do.
The answer is yes it can be done, but the some things are more complex than others. EG you want the devices to record temperature in different places, then each device will simply create a new record every few minutes along with the location name. the name and time of the record would be the unique key. No problem.
If, say, you want to be able to change any record in the database on any device, you need to think about how two people changing the same record will be reconciled.
It also depends on what you mean by "simultaneously". In general, database writes are done sequentially in "transactions". So you may need to consider whether "simultaneously" means "very quickly one after another" or not. Does the order of the writes matter?
Related
I am java developer and my application is in iOS and android.I have created web service for that and it is in restlet Framework as JDBC as DB connectivity.
My problem is i have three types of data it is called intersection like current + Past + Future.and this intersection contain list of user as a data.There is single web service for giving all users to device as his/her intersection.I have implement pagination but server has to process all of his/her intersections and out of this giving (start-End) data to device.I did this because there are chances that past user may also come in current.This the total logic.
But as intersection grows in his/her profile server has to process all user.so it become slow and this is obvious.also device call this web service in every 5 minutes.
please provide better suggestion to handle this scenario.
Thanks in advance.
Ketul Rathod
It's a little hard to follow your logic, but it sounds like you can probably benefit from caching your results on the server.
If it makes sense, after every time you process the users data on the server, save the results (to a file, to a database table, whatever). Then, in 5min, if there are no changes, simply return the same. If there were changes, retrieve from cache (optionally invalidating the cache in the process), append those changes to what is cached, re-save the results in the cache, and return the results.
If this is applicable to your workflow, your server-side processing time will be significantly less.
I have a very small program that is going to be used by two or more computers at the same time. It's a small program that can add string to the list and remove it from the list. I save all these strings in a remote postgres database. When I delete string from the list, it gets deleted from the database however if the program is running on other computer you still can see this string. Currently I have only one option in mind which is refreshing data in program every x time ? Are there better options ? Database is very small, only one column and shouldnt be more than 100 rows.
You should never allow client programs to interact with a remote database directly. Exposing your database to the clients is a huge security problem. You should always have a server-program in between which communicates with the clients, validates their input, communicate with the database, and then tells the clients what they want to know.
This would also give you the ability to add push-updates to your network protocol (when one client makes a change, update the database and also inform the other client about the change).
But when you really want to take the risk and you consider a server-program too much complexity, you could add a timestamp to every row when it gets changed. That way you could refresh the clients at regular intervals by querying only for the rows which got changed since the last refresh.
Another option would be to allow the clients to communicate with each other in a peer-to-peer manner. When a client makes a change, it doesn't just notify the database, it also notifies the other clients via separate network connections. In order to do that, the clients need to know each others IP addresses or at least each others hostnames. When these aren't known, you could have the clients write their IPs to another table of the database when they connect, so the other client can query for it. Just make sure the entries get deleted, so you don't annoy every single IP address some client ever had.
I'm developing an online chess game by Servlet and MySQL. And my program have mutil match (2 players unit).
Now I want to store each Match's data (String) on server and every player do his turn, this Match's data will be updated.
What the best solution to store this temporary data?
There's absolutely nothing wrong with hitting a database every second as long as you are using a connection pool. Use the database. If you're really afraid of that, use memcache or reddis.
Mysql can handle hundreds of queries a second on a small server.
See: https://serverfault.com/questions/164980/how-many-selects-per-second-can-a-mysql-server-run
Those are selects... but they're talking thousands. If you're just updating a row for the state of a chess game, that's a very simple write operation.
I have a scenario where my Java program has to continuously communicate with the database table, for example my Java program has to get the data of my table when new rows are added to it at runtime. There should be continuous communication between my program and database.
If the table has 10 rows initially and 2 rows are added by the user, it must detect this and return the rows.
My program shouldn't use AJAX and timers.
If the database you are using is Oracle, consider using triggers, that call java stored procedure, that notifies your client of changes in the db (using JMS, RMI or whatever you want).
without Ajax and timers, it not seems to do this task.
I have also faced the same issue, where i need to push some data from server to client when it changes.
For this, you can user Server push AKA "Comet" programming.
In coment
we make a channel between client and server, where client subscribes for particular channel.
Server puts its data in the channel when it has it.
when client reads the channel, it gets all the data in the channel and channel is emptied.
so every time client reads from channel, it will get new data only.
Also to monitor DB changes, you can have two things,
Some trigger/timer (Check out Quartz Scheduler)
Event base mechanism, which pushes data in the channel on particular events.
basically, client can't know anything happening on server side, so you must push some data or event to tell client that, i have some new data, please call some method. Its kind of notification. So please check in comet/server push with event notification.
hope this helps.
thanks.
Not the simplest problem, really.
Let's divide it into 2 smaller problems:
1) how to enable reloading without timers and ajax
2) how to implement server side
There is no way to notify clients from the server. So, you need to use flash or silverlight or JavaFX or Applets to create a thick client. If the problem with Ajax is that you don't know how to use it for this problem then you can investigate some ready-to-use libraries of jsp tags or jsf components with ajax support.
If you have only 1 server then just add a cache. If there are several servers then consider using distributed caches.
If you have a low-traffic database you could implement a thread that rapidly checks for updates to the DB (polling).
If you have a high-traffic DB i wouldn't recommend that, 'cause polling creates much additional traffic.
server notifying client is not a good idea (consider a scenario with a 1000 clients). Do u use some persistence layer or u have to stick to pure JDBC?
If you have binary logs turned on in MYSQL , you can see all of the transactions that occur in the database.
A portable way to do this, is adding a column time stamp (create date) which indicates when the row was added to the table. After initial loading of the content you simply poll for new content which a where clause current_time >= create_date. In case that rows could have identical timestamps you need to filter duplicates before adding them.
I am developing an application which will be integrated with thousands of sensors sending information at every 15 minute interval. Let's assume that the format of the data for all sensors is same. What is the best strategy of storing this data so that every thing is archived (is accessible) and does not have a negative impact due to large size of growing data.
Th question is related to general database design I suppose, but I would like to mention that I am using Hibernate (with Spring Roo) so perhaps there is some thing already out there addressing it.
Edit: sensors are dumb, and off the shelf. It is not possible to extend them. In the case of a network outage all information is lost. Since the sensors work on GPRS this scenario will be some what unlikely (as the GPRS provider is a rather good one here in Sweden, but yes it can go down and one can do nothing about it).
A queuing mechanism was foremost in consideration and spring roo provides easy to work with prototype code based on ACTIVEMQ.
I'd have a couple of concerns about this design:
Hibernate is an ORM tool. It demands an object model on one side and a relational one on the other. Do you have an object representation? If not, I'd say that Hibernate isn't the way to go. If it's a simple table mapping mechanism you'll be fine.
Your situation sounds like war: long periods of boredom surrounded by instants of sheer terror. I don't know if your design uses asynchronous mechanisms between the receipt of the sensor data and the back end, but I'd want to have some kind of persistent queuing mechanism to guarantee delivery of all the data and an orderly line while they were waiting to be persisted. As long as you don't need to access the data in real time, a queue will guarantee delivery and make sure you don't have thousands of requests showing up at a bottleneck at the same time.
How are you time stamping the sensor items as they come in? You might want to use a column that goes down to nanoseconds to get these right.
Are the sensors event-driven or timed?
Sounds like a great problem. Good luck.
Let's assume you have 10,000 sensor sending information every 15 minutes. To have better performance on database side you may have to partition your database possibly by date/time, sensor type or category or some other factor. This also depend on how you will be query your data.
http://en.wikipedia.org/wiki/Partition_(database)
Other bottle neck would be your Java/Java EE application itself. This depends on your business like, are all 150,000 sensors gonna send information at same time? and what architecture your java application gonna follow. You will have to read articles on high scalablity and performance.
Here is my recommendation for Java/Java EE solution.
Instead of single, have a cluster of applications receiving the data.
Have a controller application that controls link between which sensor sends data to which instance of application in the cluster. Application instance may pull data from sensor or sensor can push data to an application instance but controller is the one who will control which application instance is linked to which set of sensors. This controller must be dynamic such that sensors can be added or removed or updated as well application instances can join or leave cluster at any time. Make sure that you have some fail over capability into your controller.
So if you have 10,000 sensors and 10 instances of application in cluster, you have 1000 sensors linked to an application at any given time. If you still want better performance, you can have say 20 instances of application in cluster and you will have 500 sensors linked to an application instance.
Application instances can be hosted on same or multiple machines so that vertical as well as horizontal scalability is achieved. Each application instance will be multi threaded and have a local persistence. This will avoid bottle neck on to main database server and decrease your transaction response time. This local persistence can be a SAN file(s) or local RDBMS (like Java DB) or even MQ. If you persist locally in database, then you can use Hibernate for same.
Asynchronously move data from local persistence to main database. This depends on how have you persisted data locally.
If you use file based persistence, you need a separate thread that reads data from file and inserts in main database repository.
If you use a local database then this thread can use Hibernate to read data locally and insert it on main database repository.
If you use MQ, you can have thread or separate application to move data from queue to main database repository.
Drawback to this solution is that there will be some lag between sensor having reported some data and that data appearing in main database.
Advantage in this solution is that it will give you high performance, scalability, and fail-over.
This means you are going to get about 1 record/second multiplied by how many thousand sensors you have, or about 2.5 million rows/month multiplied by how many thousand sensors you have.
Postgres has inheritance and partitioning. That would make it practical to have tables like:
sensordata_current
sensordata_2010_01
sensordata_2009_12
sensordata_2009_11
sensordata_2009_10
.
.
.
each table containing measurements for one month. Then a parent table sensordata can be created that "consists" of these child tables, meaning queries against sensordata would automatically go through the child tables, but only the ones which the planner deduces can contain data for that query. So if you say partitioned your data by months (which is a date range), and you expressed that wish with a date constraint on each child table, and you query by date range, then the planner - based on the child table constraints - will be able to exclude those child tables from execution of the query which do not contain rows satisfying the date range.
When a month is complete (say 2010 Jan just turned 2010 Feb), you would rename sensordata_current to the just completed month (2010_01), create a new sensordata_current, move over any rows from 2010_01 into the newly created sensordata_current that have a timestamp in Feb, add finally a constraint to 2010_01 that expresses that it only has data in 2010 Jan. Also drop unneeded indices on 2010_01. In Postgres this all can be made atomic by enclosing it into a transaction.
Alternatively, you might need to leave _current alone, and create a new 2010_01 and move over all January rows into it from _current (then optionally vacuum _current to immediately reclaim the space - though if your rows are consant size, with recent Postgres versions there is not much point in doing that). Your move (SELECT INTO / DELETE) will take longer in this case, but you won't have to write code to recreate indices, and this would also preserve other details (referential integrity, etc.).
With this setup removing old data is as quick and efficient as dropping child tables. And migrating away old data is efficient too since child tables are also accessible directly.
For more details see Postgres data partitioning.
Is it a requirement that these sensors connect directly to an application to upload their data? And this application is responsible for writing the data to the database?
I would consider having the sensors write data to a message queue instead, and having your "write to DB" application be responsible for picking up new data from the queue and writing it to the database. This seems like a pretty classic example of "message producers" and "message consumers", i.e. the sensors and the application, respectively.
This way, the sensors are not affected if your "write to DB" application has any downtime, or if it has any performance issues or slowdowns from the database, etc. This would also allow you to scale up the number of message consumers in the future without affecting the sensors at all.
It might seem like this type of solution simply moves the possible point of failure from your consumer application to a message queue, but there are several options for making the queue fault-reliant - clustering, persistent message storage, etc.
Apache MQ is a popular message queue system in the Java world.