The main point of this project is to add a timestamp to the database each second.
What i need.
1. BackgroundTask for adding generated timestamp to db.
2. BackgroundTask for adding data to Buffer while server connection is offline.
3. Add the data that was saved to the buffer inside the db, while saving new timestamps while the app is working
I have compleated the 1st and 2nd part but having trouble figuring out the 3rd part.
I have 2 thread classes and both implement Runnable.
When server status is positive, Thread A adds data to database.
When server status is negative Thread B creates a Buffer and stores the data there.
Now I need Thread C , that tries to connect to the server every 5 seconds and when it establishes the connection Thread B should somehow implement the data to database(in FIFO order) .
I'm having trouble figuring out what to do with the threads and the correct way of implementing further functionality, could someone give me a navigation of how I should implement the following functionality?
How about using BlockingQueue and specifically LinkedBlockingQueue(FIFO)?
Thread A will keep adding the data in queue no matter the connection is available or not and thread B will try to read and commit to DB so in case of no data in queue Thread B will block and wait for data .
Note : Go for unbounded Blocking queue if you want to ensure no tasks are rejected.
I would implement it using two threads, thread A would write to a LinkedList every second, and that's all thread A would be concerned about; and thread B to continuously read from the head (LinkedList.remove()) and attempt to upload it to the database, this way if it fails you can retry indefinitely until it succeeds and then continue reading the head of the LinkedList.
However, you would have to keep thread safety in mind. That being said, I think you should be fine if you just set thread B to run half a second behind thread A since thread B would never get ahead of thread A, even if every single upload to the database is successful.
Related
Here's the scenario:
ThreadA is going to read from some socket, and write data to "MyFile.txt"
ThreadB is going to read "MyFile", and when it reaches the end, it will loops until new data are available in MyFile (because i don't want to re-open "MyFile.txt", and lose the time so i reach the position from where i was..).
Is it possible to do such a thing ?
If not, is there another way to do such a thing ?
The problem you mention is a famous Producer Consumer Problem
Common solution to this is to use BlockingQueue
An example of real world usage is in AjaxYahooSearchEngineMonitor
What Thread A does is, it will submit a string to queue, and then return immediately.
What Thread B does is, it will pick up the item from queue one by one, and process them.
When there is no item in the queue, Thread B will just wait there. See line 83 of the source code.
I have a traditional problem that's living in java development for years, but couldn't decide what is the best way of doing it. So need your advice to pick the best one. Issue goes as below -
Client - Java program (Basically a web based application)
Server - Written in C
Requirement - User will upload a file that may contain a million records or for example say 50000 records. Each record (line) will have a serial id, product name and customer name. My java program should read the file and send request to the C application over the network. This C server application will respond with a request id and in java (client) I need to store this request ID in a list that's synchronized and should query back to the C server application to find out the status of the request id that was sent earlier. And server responds either WIP (work in progress) or DONE (completed) for the request id. If C server application response = DONE, server will send some data along with the response and if its WIP, client should retry 3 times with an interval of 5 seconds.
Code Design -
Step 1 - Read the file line by line
Step 2 - After reading the line, Start a thread that would send request to the server and stores the response to a synchronized array list.
Step 3 - Another thread would read the synchronized list and starts querying the request status and stores the final response from the C server app.
This might create a memory overhead as you can see if the file contains 100000 records, it might create 100000 threads. Can you guys suggest me a better way of handling this.
Thanks, Sirish.
The simple solution is: don't create 100000 Threads, but use an ExecutorService (eg use one of the ready to use from Executors).
That way you can control the maximum number of parallel jobs to be executed.
Several issues/questions/suggestions -
A. Are the records in the file dependent on each other - if not, what you can do is probably do is split the file into files, and have each thread perform Step 1.
B. Don't start threads after reading each line - use thread pools and/or have worker threads that listen to queue for "jobs" to do - push each line your read on such a queue. This will save you the thread start time.
C. I would consider having a different data structure if possible - maybe not just single list (once again, if the lines are not dependent on each other, this is possible) - pay attention that when you're locking the list you're locking the entire list.
If you do want to work with a single list, and the lines are not dependant on each other - consider having N threads reading from the list, and have a ReaderWriterLock - this way, the N reading threads can work (if they perform just read, and are not responsible for deleting the entry they read) while the writer thread fills the list.
Can you help me in two problem :
A. We have a table on which read and write operation happens simultaneously. Write happens very vastly so read is very slow - sometimes my web application does not come up due to heavy write operation on this table. How could i handle such scenario. Write happens through different Java application while read happens through our web application, so web application become very slow. Any idea?
B. Write happens to this table happens through 200 threads, these thread take connection from connection pool and write into the table and this application run 24 by 7. is the thread priority is having issue and stopping read operation from web application.
C. Can we have master- master replication for that table only- so write happens in one table and write happens in other table and every two minute data migrates from one table to other table?
Please suggest me .
Thanks in advance.
Check connection pool size - maybe it's too small and your threads waste time waiting for connection from pool.
Check your database settings, if you just running it with out-of-the-box params there maybe a good space for improvements.
You probably need some kind of event-driven system - when vehicle sends data DB is not updated, but a message is added to some queue (e.g. JMS). Your app then caches data on startup, and updates both cache and database upon receiving this message. The key thing is that the only component that interacts with DB is your app, and data changed only when you receive event - so you don't need to query DB to read the data, plus you may do updates in the background using only few threads, etc. There are quite good open-source messaging systems (e.g. Apache Active MQ) and caching libraries (e.g. EH Cache), so you can built reasonably perfomant and fault-tolerant system with not too much effort.
I guess introducing messaging will be a serious reengineering, so to solve your immediate problem replication might be the best solution - merge data from the updateable table to another one every 2 minutes, and the tracker will read that another table; obviously works well if you only read the data in the web-app, and not update them, otherwise you need to put a lot of effort to keep 2 tables in sync. A variation of that is batching - data from vehicle are iserted into intermediate table, and then every 2 minutes transferred into main table from which reader queries them; intermediate table is cleaned after transfer.
The one true way to solve this is to use a queue of write events and to stop the writing periodically so that the reader has a chance.
Create a queue for incoming write updates
Create an atomicXXX (see java.util.concurrency) to use as a lock
Create a thread pool to read from the queue and execute the updates when the lock is unset
Use javax.swing.Timer to periodically set the lock and read the table data.
Before trying anything too complicated try this perhaps:
1) Don't use Thread priorities, they are rarely what you want.
2) Set up your own priority scheme, perhaps simply by having a (priority) queue for both reads and writes where reads are prioritized. That is: add read and write requests to a single queue and have them block or be notified of the result.
3) check your database features to optimize write heavy tables
i have the following situation:
Read data from database
do work "calculation"
write result to database
I have a thread that reads from the database and puts the generated objects into a BlockingQueue. These objects are extremely heavy weight hence the queue to limit amount of objects in memory.
A multiple threads take objects from the Queue, performs work and put the results in a second queue.
The final thread takes results from second queue and saves result to database.
The problem is how to prevent deadlocks, eg. the "calculation threads" need to know when no more objects will be put into the queue.
Currently I achieve this by passing a references of the threads (callable) to each other and checking thread.isDone() before a poll or offer and then if the element is null. I also check the size of the queue, as long as there are elements in it, the must be consumed. Using take or put leads to deadlocks.
Is there a simpler way to achieve this?
One of the ways to accomplish would be to put a "dummy" or "poison" message as the last message on the queue when you are sure that no more tasks are going to arrive on the queue.. for example after putting the message related to the last row of the db query. So the producer puts a dummy message on the queue, the consumer on receiving this dummy message knows that no more meaningful work is expected in this batch.
Maybe you should take a look at CompletionService
It is designed to combine executor and a queue functionality in one.
Tasks which completed execution will be available from the completions service via
completionServiceInstance.take()
You can then again use another executor for 3. i.e. fill DB with the results, which you will feed with the results taken from the completionServiceInstance.
Here's the scenario:
ThreadA is going to read from some socket, and write data to "MyFile.txt"
ThreadB is going to read "MyFile", and when it reaches the end, it will loops until new data are available in MyFile (because i don't want to re-open "MyFile.txt", and lose the time so i reach the position from where i was..).
Is it possible to do such a thing ?
If not, is there another way to do such a thing ?
The problem you mention is a famous Producer Consumer Problem
Common solution to this is to use BlockingQueue
An example of real world usage is in AjaxYahooSearchEngineMonitor
What Thread A does is, it will submit a string to queue, and then return immediately.
What Thread B does is, it will pick up the item from queue one by one, and process them.
When there is no item in the queue, Thread B will just wait there. See line 83 of the source code.