I am working on an Android app which purpose is to download chunks(parts of a video file) from 2 servers, append them in order(into a main video file) after each one is downloaded and finally play this video file while downloading continues..
This works well when downloading is done serial by using two different threads(one for each server) that perform downloading. I want to know how it is possible to achieve the above but with concurrent downloading, instead of serial.
That is to download chunks from servers at the same time and in order. For example, for the same period of time download chunk0, chunk1 & chunk2 from server1 (which let's say is 3 times faster than server2) and chunk3 from server2, so that we totally use all the available bandwidth of the 2 servers at this period of time. And this process repeats until all chunks are downloaded.
By using threads and join, downloading is serial, as i said above. In order to make it concurrent, i tried to remove join from each thread, but then it doesn't download chunks in order and also download only from one server, not from both. AsyncTask is not a solution, as it also doesn't download chunks in order.
So, is there any way to achieve this concurrent and in order downloading of chunks as I described it above? Has anyone done something like this as a project, so as to know an answer for sure?
You may use the popular technique among download accelerators.
In general, the idea is about requesting chunks from each server using the Range HTTP header. (The server responds the Accept-Ranges header when it is capable of processing the Range header accordingly). (This blog has a good explanation about that).
Every thread/runnable/callable has to know which chunk is its responsibility ( first byte position + length ?) because each one will have to write its own part in the file.
Then there will be a decision to be made, you can:
Write the file using an instance of RandomAccessFile in each thread, obviously positioning the file pointer in the first byte position of its chunk (with the seek method), or..
Be sure that you have a unique worker thread (see Executors and submit) that is in charge of writing the bytes told by each thread. As in the moment of writing, you will use seek to move the file pointer into the correct position, there will be no errors of overlapping.
NOTE: If you want to be able to start the playback when you have your first chunk, you may do it by executing that code after the first chunk thread download+write has finished.
Related
I'm building a library (java 8) that needs to listen to the modifications of several files. The library needs parse the newly added lines in the corresponding files each time a modification occurs.
The files are like event logs. So they are always appended (no deletion or overriding)
I have two questions:
Is there a way to know what are the newly added lines in a file when its modified? (is there a functionality in java NIO to identify this?)
I've seen solutions in NIO package (Watch Service API) that can be used as poll based mechanism to listen to file modification. Is there another native solution to make it push based? so that I don't need to keep polling between intervals of time.
I'm mainly looking for a native solution but third party solution suggestion are also appreciated
thanks
You implement "push" by running a take() loop in its own thread. On each take() you start a new thread to process the changed lines.
To determine what changed, keep track of the file length each time... Changed data is at offset previousSize for length newSize - previousSize.
You'll probably want some kind of event coordination to prevent starting the process again before the previous instance completes. That could happen if there are multiple changes in a very short period of time.
I have a problem caused by multi-threading and Android Open Accessory.
I need to communicate with a USB Accessory, but I need to do it from 2 threads. One thread generates and sends data the other one reads data.
Why I don't use a single thread? Because there can be 1 or more writes before a read and reads are blocking, so that is not an option.
If using multiple threads, I do run into "I/O Error (No such device)" sooner or later, because I will have a collision between read & write being executed at the same time.
Locking will more or less put me back in single-thread situation, so not good.
.available() method on the input-stream returns is not supported, so I cannot check if anything is available before doing a read
Since it's not a socket-based stream I cannot set timeout either.
I have tried getting the FileDescriptor from the USBAccessory and passing to JNI to handle it there, but after the first read/write the device becomes inaccessible.
Question/Suggestion needed:
What will be a suggested/best-practice approach to this? I do not expect written code, I just need some guidance on how to approach this problem.
To clarify:
The software at the other end might or might NOT respond with any data. There are some so called silent sends were the data sent it's just received but there is no ACK. Since the app I'm working on is only a proxy, I do not have a clear picture if the data will or will not produce an answer. That will require analysis of the data as well, which isn't on the books at the moment.
As you want to do read and write in parallel, writing will always lead to a pause to read if the read is on the same part as write.
May be you can follow similar approach as ConcurrentHashMap and use different locks for different segments and lock read only if write is on the same segment else allow the read to happen.
This will
Avoid blocking read during write in most scenarios
Avoid collision and
Definitely wont be a single thread approach.
Hope that helps.
If using multiple threads, I do run into I/O Error (No such device)
sooner or later, because I will have a collision between read & write
being executed at the same time.
This says it all. Since you are doing read and write on the same channel that does not support concurrent access, you are required to have your thread wait until the other thread is done doing read/write.
Your two-thread approach is what I would do, more or less. Good luck and trust in yourself.
I have two (Java) processes on different JVMs running repeatedly. The first one regularly finds some "information" and needs to store it somewhere. The second process regularly reads this information to handle it. The intervals are more or less random, so process 1 may find three pieces of information until process 2 reads them or vice versa.
My approach is to write this information to text files. But I am afraid that appending and reading the text files accidentally happens at the same time so that I run into locks. But writing a new text file for each piece of information seems like overkill.
What would be a better solution?
EDIT: I am sorry, I did not make clear: The java processes run in different JVMs. They cannot see each other directly.
You can get this to work, provided you are careful with file handling and you don't have a high update rate e.g. 10 updates per second.
Note: you could do it with file renaming instead of locks.
What would be a better solution?
Just about anything, SO is not for recommending things, but in this case I could recommend just about anything without more specific requirements. I could for example recommend my library Chronicle Queue because I wrote it and I sure it could do what you want, however there are many possible alternatives.
I am sending about one line of text every minute.
So you can write a temporary file for each message, rename it when finished. The consumer can have a directory watcher so it knows as soon as you have done this. The consumer could delete the file when done. This has an overhead but it would be less than 10 ms.
If you want to keep a record of all messages, the producer can also write to a log file.
I have a scenario where I need to process a csv file that contains some simulation data from a device. Each line is an output representing the device state at a point in time. On processing each line, specific columns are checked for variance / anomalies.
If there are anomalies, an email has to be sent to a bunch of folks with the detected anomaly. However to avoid spamming them (csv can occasionally be several 100 thousand lines) I have to maintain a threshold of X seconds.i.e If a mail was sent for the same anomaly from the same condition (from the same device being simulated) < X seconds back, I must just ignore sending the mail.
Currently the solution I use seems clumsy to me, where
1) I save the mail-message and device id with anomaly detection time.
2) Create one "alert" per email-id with a create-time-stamp, sent-time-stamp, message-id (from step 1) and device-id with status as "NEW".
3) Before sending each mail I do a database read to see if the last email with status as 'SENT' has a time stamp that exceeds the threshold to ignore. ( now - sent-time-stamp > threshold)
If yes, then I get all the alerts using the message-id and send them out and update all their status to SENT- else just ignore.
I started off with a thread pool executor and realized halfway through that the read-send condition can fail once there are multiple threads trying to send out emails and update the sent-time-stamp. So for now I have set the thread pool size to 1 - which beats the purpose of an executor. (I don't have row level locking as I use Mongo as the backing db). The backing datastore has to be a nosql store as the fields can vary drastically and will not fit a machine's disk as more simulations get piped in.
The application is distributed - so a csv file can be picked by any random node to process and notify.
Would Akka be a good candidate for this kind of process ? Any insights or lessons from prior experience implementing this are welcome (I have to stick with JVM).
You can use distributed Akka as replacement (see good tutorial here http://www.addthis.com/blog/2013/04/16/building-a-distributed-system-with-akka-remote-actors/#.U-HWzvmSzy4) but why? Just bit update what already works:
1) Remove Executor at all, it's not needed here, send emails one by one (I suppose you're not trying to send millions of mail messages at once, right?)
2) Cleanup database for old messages on application start to resolve problems with disk space.
Akka could help you with the distribution if you use Akka Cluster. That gives you a dynamic peer-to-peer cluster on your nodes, very nice if you need it. FApart from that, Akka works message-based which sounds like a good match to model your domain.
However, be aware that Akka bases on the actor programming model, which is great but really different from multi-threaded programs in java. So there is a learning curve. If you need a quick solution, it will probably not be the best match. If you are willing to put some time into this and learn what Akka is about, it could be a good match.
I have stuck in a serious problem. I am sending a request to server which contains some URL as its data.If I explain it , it is like I have a file which contains some URL in a sequential order I have to read those sequential data by using thread. Now the problem is there are one hundred thousand URL, I have to send each URL in the server in a particular time(say suppose 30 seconds).So I have to create threads which will serve the task in the desired time. But I have to read the file in such a way if first thread serve first 100 URL then 2nd thread will serve the next 100 URL and in the same way the other threads also.And I am doing it in a socket programming,so there is only one port at a time that I can use. So how to solve this problem. Give me a nice and simple idea, and if possible give me an example also.
Thanks in Advance
Nice and simple idea (if I understand your question correctly): You can use a LinkedList as a queue. Read in the 1,000 urls from file and put them in the list. Spawn your threads, which then pull (and remove) the next 100 urls from the list. LinkedList is not thread-safe though, so you must synchronize access yourself.
One thing that you could look into is the fork/join framework. The way that the java tutorials explains this is: "It is designed for work that can be broken into smaller pieces recursively. The goal is to use all the available processing power to make your application wicked fast". Then all you really need to do is figure out how to break up your tasks.
http://download.oracle.com/javase/tutorial/essential/concurrency/forkjoin.html
you can find the jar for this at: http://g.oswego.edu/dl/concurrency-interest/