(please read UPDATE 3 at the end)I'm developing an app that continually works with the sensors of device, works with Accelerometer and Magnetic sensors to retrieve the orientation of device(the purpose is mentioned here). in other words, my app needs to know the orientation of device in Real-time(however this is never possible, so as fast as possible instead, but really as fast as possible !). as mentioned in professional Android 4 Application Development by Reto Meier:
The accelerometers can update hundreds of times a second...
I must not lose any data that sensors report and I also want to do time-consuming operations on these data(retrieve the orientation and then do calculations... ). I decided to solve my problem by using LinkedBlockingQueue:
public void startSensors() {
LinkedBlockingQueue<float[][]> array=new LinkedBlockingQueue();
sensorListenerForOrientation = new SensorEventListener() {
#Override
public void onSensorChanged(SensorEvent event) {
if (event.sensor.getType() == Sensor.TYPE_ACCELEROMETER)
aValues = (event.values.clone());
else if (event.sensor.getType() == Sensor.TYPE_MAGNETIC_FIELD)
mValues = (event.values.clone());
if (aValues != null && mValues != null) {
try {
array.put(new float[][] { aValues, mValues });
} catch (InterruptedException e) {
}
}
}
#Override
public void onAccuracyChanged(Sensor sensor, int accuracy) {
}
};
Sensor aSensor = sm.getSensorList(Sensor.TYPE_ACCELEROMETER).get(
sm.getSensorList(Sensor.TYPE_ACCELEROMETER).size() - 1);
Sensor mSensor = sm.getSensorList(Sensor.TYPE_MAGNETIC_FIELD).get(
sm.getSensorList(Sensor.TYPE_MAGNETIC_FIELD).size() - 1);
sm.registerListener(sensorListenerForOrientation, aSensor,
SensorManager.SENSOR_DELAY_FASTEST);
sm.registerListener(sensorListenerForOrientation, mSensor,
SensorManager.SENSOR_DELAY_FASTEST);
executor.execute(new Runnable() {
#Override
public void run() {
doCalculations();
}
});
}
and
public void doCalculations() {
for (;;) {
float[][] result = null;
try {
result = array.take();
} catch (InterruptedException e) {
}
float[] aValues, mValues;
aValues = result[0];
mValues = result[1];
int[] degrees=getOrientation(aValues,mValues);
Log.e("",String.valueOf(degrees[0]));
//other calculations...
}
}
now I pick up my device and rotate it about 90 degrees to right and then return it to the first position fast(for example in 1.5 seconds) but as I look at the orientations that are registered in device I see for example: 0,1,2,3,4,5.......,40,39,38,37,....,0
I just want to say that I can't see a large domain of degrees in my result .
based on what I have done and what I have researched I just can be sure that I am NOT losing any data, any new data reported by sensors are recorded.
any Idea, solution?!
Regards!
UPDATE 1: I did another experiment with my device and got shocking results! if I rotate my device over an axis 90 degrees fast (less than a second), I can see all degrees in my result: 0,1,2,3,....,89,90 (for example) but if I rotate it 90 degrees and then rotate it back to its first position, the result would be 0,1,2,...,36,37,36,...2,1,0(for example)...really confusing !
UPDATE 2: I updated doCalculations() method to be more clear what I have done
UPDATE 3: I think maybe we can solve the problem in another way! I have clear purposes for this code. please have a look at this. I
have mentioned what is going to happen, I need to detect an specific
movement gesture. so maybe the whole way that I have chosen(the
technique above) is not a good way for solving this problem. maybe
it's better to detect that gesture by using other sensors or using the
same sensors in other way. what do you think?
So it looks like you are trying to find high throughput low latency solution for a standard "Producer-Consumer" problem. Basically the idea is quite straightforward: decrease data handling overhead, process data in parallel. Suggestions are the following:
1. Use "low latency" libraries
javolution.org - is a real-time library aiming to make Java or Java-Like/C++ applications faster and more time predictable. It includes Android support.
mentaqueue - is a super-fast, garbage-less, lock-free, two-thread (producer-consumer) queue based on the Disruptor ideas. Android support is undefined (it looks like it should work).
disruptor - yet another lightning fast library
trove - provides high speed regular and primitive collections for Java.
Any of these solution will let you save a lot of CPU cycles.
2. Process data wisely
There is an overhead every time you submit a job. Batch processing can be really helpful.
Process data continuously. Note, executor.execute will consume quite a lot. Several long-living consumers might help.
3. Finally, use micro optimization techniques
For example, get rid of if-else-if in favor of switch.
Track performance all the time in order to identify good and bad solutions. Experiment.
Happy coding.
Just thinking: please try the following:
public void startSensors() {
final Stack<Runnable> mStack = new Stack<Runnable>();
sensorListenerForOrientation = new SensorEventListener() {
#Override
public void onSensorChanged(SensorEvent event) {
if (event.sensor.getType() == Sensor.TYPE_ACCELEROMETER)
aValues = (event.values.clone());
else if (event.sensor.getType() == Sensor.TYPE_MAGNETIC_FIELD)
mValues = (event.values.clone());
if (aValues != null && mValues != null) {
mStack.push(new Calculater(new float[][] { aValues, mValues });
}
}
#Override
public void onAccuracyChanged(Sensor sensor, int accuracy) {
}
};
Sensor aSensor = sm.getSensorList(Sensor.TYPE_ACCELEROMETER).get(
sm.getSensorList(Sensor.TYPE_ACCELEROMETER).size() - 1);
Sensor mSensor = sm.getSensorList(Sensor.TYPE_MAGNETIC_FIELD).get(
sm.getSensorList(Sensor.TYPE_MAGNETIC_FIELD).size() - 1);
sm.registerListener(sensorListenerForOrientation, aSensor,
SensorManager.SENSOR_DELAY_FASTEST);
sm.registerListener(sensorListenerForOrientation, mSensor,
SensorManager.SENSOR_DELAY_FASTEST);
new Thread() {
public void run() {
while(true)
{
try {
Runnable r = mStack.pop();
r.run();
} catch(Exception ex){}
}
}
}.start();
}
private class Calculater implements Runnable {
float[][] theValues;
public Calculater(float[][] values) {
theValues = values;
}
public void run() {
int[] degrees= getOrientation(theValues[0], theValues[1]);
Log.e("",String.valueOf(degrees[0]));
}
}
Your code looks reasonable. A big unknown is how good the sensors and sensor fusion are in your device. Quick angle change readings rely on integration of angular acceleration or else a physical gyroscope with magnetic data mixed in to make the result absolutely align with the earth. Magnetic data are subject to surroundings. If your device has low quality sensors or there are magnetic disturbances in your environment, it's entirely possible to see the kinds of error you are seeing. Big metal structures and magnetic equipment (like motors or even fluorescent light ballasts) can blank the field or introduce arbitrary errors. For normal uses, a device only needs an accelerometer to accurately determine which way is down so screen flips are accurate. This only needs to work when the device is not moving, where a gyro has no role. If you have a phone or tablet with sensors meant only to serve this purpose - therefore with no gyro or an inaccurate one - you are seeing a device limitation. The erratic values are other evidence that your device is low quality and/or that you are in a location where the earth's magnetic field is being distorted. Try the program on another (preferably expensive) device outside and in the open, and see what you get.
The usual thing to do within an event block is to do almost nothing, since this is really fast.
"Almost" being the important word. In your case, the event could just add the data of the event (from the event parameter) to some data structure (list, stack, circular buffer... your pick). That way you should lose less events (if any).
Which means that you can then (for instance periodically) read the stored events and decide if a gesture was made. That means that your intensive calculations are made less often. But you don't lose any events. I think this is acceptable because of your purpose, which is gesture recognition. I assume it doesn't have to be that fast (ie. you don't have to calculate it every time the sensor updates).
Note : this is one common way to handle IT in the Linux world.
just a thought. I have a similar problem when I needed to collect several large sample sizes an perform calculations. My situation was probably quite different from yours as I just needed acceleration. What I did was create an array list. calculated acceleration per every record reported :
#Override
public void onSensorChanged(SensorEvent event) {
float x = event.values[0];
float y = event.values[1];
float z = event.values[2];
float acceleration = FloatMath.sqrt((x * x) + (y * y) + (z * z));
Then in the same onSensorChanged method, I wait until the size hits a certain limit, like 300, clone that sample to a new list,clear out original, perform calculations on new list and continue in that manner. I get results in secs. I am not sure how much down time is allowed for your application but when I run this I get what I am looking for in less that 5 secs. If you need more sample code let me know, but that is the gist. Sorry if I didn't understand your question properly but I think you were asking for a way to calculate data without losing much? Also I have this running on a separate handler when I register the listener, not to interfere with the main thread, not to effect user experience.
Change variable declaration:
List<float[][]> array = Collections.synchronizedList(new ArrayList<float[][]>());
Inside the runnable:
Iterator<float[][]> values = array.iterator();
while (values.hasNext()) {
float[][] result = values.next();
//calculating.
//after calculating remove the items.
values.remove();
}
This is what's wrong with your code. Fast as possible requires fast coding techniques. Save the sensor type instead of evaluating it twice.
#Override
public void onSensorChanged(SensorEvent event) {
int i = event.sensor.getType();
if (i == Sensor.TYPE_ACCELEROMETER)
aValues = (event.values.clone());
else if (i == Sensor.TYPE_MAGNETIC_FIELD)
mValues = (event.values.clone());
}
Related
I would like to preface this by claiming that I am very new to android studio and java (and even this level programming in general). I am working on a simple audio synthesis application, and I was testing how multiple threads would work so that I could synthesize music and hold down multiple keys at once rather than pressing a key, calculating the entire audio buffer, and playing it once.
Here's the setup: I have 25 keys (view/buttons), that I can detect when they are pressed down and lifted up in my main UI thread. My second thread then constantly runs in the background and constantly calculates the audio buffer, and passes it to the AudioTrack stream to play. If no keys are pressed, it simply passes it a zero buffer so that nothing gets played essentially.
Here is the problem: I have set up a volatile array to represent which keys are pressed, so that when a button gets pressed in my app on the tablet, it sets an appropriate flag in the boolean array. My second thread then should be able to use that array to further calculate the audio buffer. Currently, I have it setup for 25 keys, but have only linked 2 keys in my UI, so it only recognizes when 2 of them are pressed, and the other 25 are simply set to zero. The second thread is able to recognize when either one or both of the keys are pressed, but there is a lag. In my main UI thread, I have set it up where it changes the color of the key when it detects a touch event, and that responds immediately, but the sound playing starts after a while. However, I can tap the key quickly, and it will detect the two events such as touch down and release (indicated by the keys changing color), but its as if the second thread did not recognize that I pressed the key, as it does not play the sound for even a brief moment (milliseconds). It seems like there is a few hundred millisecond lag between the key press being registered between the two threads. However, I think that the volatile boolean array keyPress does change the variable quickly, because if it didn't, then even with the lag, the second thread would play the sound for a brief second. But rather, it seems like the main UI thread changes the appropriate keyPress array element to true when a key is pressed, and also sets it to false when I lift my finger, but I can somehow do this without the second thread recognizing this happened? How is this possible? Is the main thread not actually changing the volatile array properly quick enough? Like it recognizes that a key is pressed, changes the local cached copy of the variable, and then changes the volatile variable? What would cause this lag? Is it based on the nature of the loops I have formed in the second threadLoop() thread? I don't think the thread is too slow for computing a sound buffer, because it doesn't break up the sound between buffers, so I don't think that is the problem.
I should also mention that I am not sure if "volatile" is the best variable type to use here either, so if there's some other data type, or something else I should be using, please let me know. I looked a little bit into Handlers and Loopers, but I thought that wasn't required here because those queue the tasks while I want keyPresses to be detected by the thread responsible for playing audio instantly (and also because it was kinda hard for me to wrap my head around them so I didn't bother with it, but if you think I am doing it incorrectly with this method of using a thread, please let me know). I am also using a Lenovo tab M10 in case that is relevant information. If there is any more information you would like from me, please let me know, and if a similar post already exists (I couldn't find something that answered my exact question), please point me in the right direction. Thank you very much for your help!
public class MainActivity extends AppCompatActivity {
private boolean isActive = false;
private Thread audioThread;
volatile boolean[] keyPress = new boolean[25];
//volatile boolean keypress;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
Arrays.fill(keyPress, false);
//keypress = false;
final View c4 = (View) findViewById(R.id.c4);
final View d4 = (View) findViewById(R.id.d4);
c4.setOnTouchListener(new View.OnTouchListener() {
#Override
public boolean onTouch(View view, MotionEvent motionEvent) {
int action = motionEvent.getAction();//.getActionMasked(motionEvent);
if (action == MotionEvent.ACTION_DOWN) {
c4.setBackgroundColor(Color.parseColor("#ed1b24")); //"#81DAF5"
keyPress[0] = true;
//keypress = true;
//audioThread.start();
}
else if (action == MotionEvent.ACTION_UP) {
c4.setBackgroundColor(Color.parseColor("#FFFFFF"));
keyPress[0] = false;
//keypress = false;
//audioThread.stop();
}
return true;
}
});
d4.setOnTouchListener(new View.OnTouchListener() {
#Override
public boolean onTouch(View view, MotionEvent motionEvent) {
int action = motionEvent.getAction();//.getActionMasked(motionEvent);
if (action == MotionEvent.ACTION_DOWN) {
d4.setBackgroundColor(Color.parseColor("#ed1b24")); //"#81DAF5"
keyPress[1] = true;
//audioThread.start();
}
else if (action == MotionEvent.ACTION_UP) {
d4.setBackgroundColor(Color.parseColor("#FFFFFF"));
keyPress[1] = false;
//audioThread.stop();
}
return true;
}
});
new Thread(new Runnable() {
#RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
#Override
public void run() {
threadLoop();
}
}).start();
}
#RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
private void threadLoop(){
AudioTrack audioTrack;
int intBufferSize;
short[] shortAudioData;
int intSampleRate = AudioTrack.getNativeOutputSampleRate(AudioManager.STREAM_MUSIC);
intBufferSize = AudioTrack.getMinBufferSize(intSampleRate, AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT);
shortAudioData = new short[intBufferSize];
audioTrack = new AudioTrack(AudioManager.STREAM_MUSIC,
intSampleRate,
AudioFormat.CHANNEL_OUT_MONO,
AudioFormat.ENCODING_PCM_16BIT,
intBufferSize,
AudioTrack.MODE_STREAM);
audioTrack.setPlaybackRate(intSampleRate);
int n = 0;
float x;
float TS = 1.0f/intSampleRate;
float freq = 440.0f;
float PI = 3.14159f;
float omega = 2.0f*PI*freq*TS;
int numKeysPressed;
int incrementer = shortAudioData.length;
audioTrack.play();
while(true){
for(int i = 0; i < incrementer; ++i){
if(n == Integer.MAX_VALUE | n < 0){
n = 0;
}
x = 0;
numKeysPressed = 0;
for(int a = 0; a < 25; ++a){
if(keyPress[a] == true) {
x += (float) (Math.sin((float) (omega * n*(a+1))));
numKeysPressed++;
}
}
x = (float)(x/numKeysPressed);
++n;
shortAudioData[i] = (short)(x * Short.MAX_VALUE);
}
audioTrack.write(shortAudioData, 0, shortAudioData.length);
}
//audioTrack.stop();
}
}
I think this could be the problem:
volatile boolean[] keyPress = new boolean[25];
What that declares is a volatile variable containing a reference to an array. The volatile semantics apply to to the variable that contains the reference. Not to the array that it refers to.
That means that when you assign to a cell of the array that keyPressed refers to, there is not going to be an write to the (volatile) variable itself. Therefore, there is no cache flush (or whatever) to guarantee that another thread will see the update to the array cell.
(A more technically accurate explanation entails the discussion of the happens before relationships, but the outcome is the same.)
One alternative would be to use an atomic array class. For example, you could use an AtomicIntegerArray where each int encodes a truth value:
// (This needs to be 'final' ... to guarantee that the keypress
// variable is visible without any additional synchronization.)
final AtomicIntegerArray keyPress = new AtomicIntegerArray(25);
// To test if a flag is set:
if (keyPress.get(i) != 0) {
// it is set.
}
// To set a flag (atomically)
keyPress.set(i, flag ? 0 : 1);
And you can do other things like atomically toggling or atomically incrementing (a counter) by using other api methods.
An AtomicReferenceArray<Boolean> would also work. I'll leave you to figure out the details.
TL;DR version
I need to scan and get the BSSID / M.A.C address and Signal Level of the nearby Access Points several times a second. WifiManager.startScan() scans about 3.5 seconds which is a bit slow. (better than 6 seconds for others but still too slow for me).
I'm looking for a way to get results every 200ms or maybe even less.
The more detailed version.
I'm trying to build a location finding app that uses WiFi signals instead of the GPS. So far so good with the exception that i need to get a lot of data in order to have accurate and consistent results.
I need to get the BSSID and the Signal Level from every Access Point I can find and than store this data for later use.
I've tried using the WifiManager and the BroadcastReceiver to scan and get the scan results. The problem with them is that they are really slow. It takes at least 3 seconds for a single scan. In 3 seconds signal level value will change , leading to inaccurate results. I also need to have several thousand scans which will take an awful loooot of time. So far I haven't found an alternative.
My phone is currently running Android 8.0.0
Perhaps if I optimise my code the scan will perform faster?
onCreate method:
#Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_sensor);
btnMainActivity = (Button) findViewById(R.id.button_ToMainActivity);
btnGetWifiInfo = (Button) findViewById(R.id.button_GetWifiInfo);
textWifiInfo = findViewById(R.id.textView_wifiInfo);
textWifiNr = findViewById(R.id.textView_wifiNr);
mWifiManager = (WifiManager) getApplicationContext().getSystemService(Context.WIFI_SERVICE);
mWifiReceiver = new BroadcastReceiver() {
#Override
public void onReceive(Context c, Intent intent) {
getScanResultInfo();
timeDifference = SystemClock.elapsedRealtime() - startTime;
textWifiInfo.setText("Seconds elapsed: "+Double.toString(timeDifference /1000.0));
nrOfScans++;
if (nrOfScans < 10){
mWifiManager.startScan();
}
}
};
getWifiInfo();
toMainActivity();
}
public void getScanResultInfo(){
int level;
List<ScanResult> wifiScanList = mWifiManager.getScanResults();
textWifiNr.setText("Nr of detected APs: "+ wifiScanList.size());
for (ScanResult scanResult : wifiScanList) {
level = WifiManager.calculateSignalLevel(scanResult.level, 5);
Log.d("WIFI","Level is " + level + " out of 5 " + scanResult.level + " on " + scanResult.BSSID + " ");
}
}
And when I press the button the scan starts.
public void getWifiInfo(){
btnGetWifiInfo.setOnClickListener(
new View.OnClickListener() {
#Override
public void onClick(View v) {
finePermission = false;
startTime = SystemClock.elapsedRealtime();
nrOfScans = 0;
checkPermissions();
if ( finePermission == true){
((WifiManager) getApplicationContext().getSystemService(WIFI_SERVICE)).startScan();
mWifiManager.startScan();
}
else {
Log.d("WIFI"," Missing Permissions: "+finePermission);
}
}
}
);
}
Thank you in advance for your time!
You can skip passive channel scanning by using the hidden API, startscanActive.
startScanActive is not an exposed API in android reference; so use it at your own risk. If you must use it, refer this - https://github.com/mozilla/MozStumbler/issues/40
Also, note that if the API indeed scans only active channels as the name suggests, you will not be able to get APs present on passive channels in your scan results
Bad news from official Android side
https://android.googlesource.com/platform/frameworks/base/+/478ee74428e04522d6ac0bf10c30ddaaf62028a4
and
https://android.googlesource.com/platform/frameworks/opt/net/wifi/+/4f11976612567e57eefb0b58c7aef1059f52b45c
announce (or propose?) to significantly limit scanResult requests per time from v9 Pie on.
In concrete values, your app should be limited to max 4 scanResult requests in 2 minutes.
This will be a disaster/nightmare for all existing WiFi-Tool apps.
Official reason: safe battery.
Let's unite our voices and let them know we want Android as a Swiss Army knife and not let it become another white brick.
I recently embarked on a project to simulate a collection of stellar bodies with the use of LWJGL. The solution required many loop iterations per frame to accomplish. The program calculates the forces exerted on each body by every other body. I did not wish to implement any form of limitations, such as tree algorithms. The program itself is able to simulate 800 bodies of random mass (between 1 and 50) at around 15 fps. Here is the original code for calculating, then updating the position of each body.
public void updateAllBodies() {
for (Body b : bodies) {
for (Body c : bodies) {
if (b != c) {
double[] force = b.getForceFromBody(c, G);
b.velocity[0] += force[0];
b.velocity[1] += force[1];
b.velocity[2] += force[2];
b.updatePosition();
}
}
}
}
Recently I came across the subject of parallels and streams. Seeing that my original code used only one thread, I thought I might be able to improve the performance by converting the array to a stream, and executing it with the use of
.parallelStream()
I don't know much about multi-threading and parallelism, but here is the resulting code that I came up with.
public void updateAllBodies() {
Arrays.asList(bodies).parallelStream().forEach(i -> {
for(Body b: bodies){
if (i != b){
double[] force = i.getForceFromBody(b, G);
i.velocity[0] += force[0];
i.velocity[1] += force[1];
i.velocity[2] += force[2];
i.updatePosition();
}
}
});
}
Unfortunately, when executed, this new code resulted in the same 15 fps as the old one. I was able to confirm that there were 3 concurrent threads running with
Thread.currentThread().getName();
At this point, I have no idea as to what the cause could be. lowering the number of bodies does show a drastic increase in frame rate. Any help will be greatly appreciated.
I cant seem to find a way to mark a comment as the answer to a post, so I will state that the best answer was given by softwarenwebie7331.
ok so I did my research there is plenty of questions here on thread synchronization but non of them really hit the point. I am currently working in Opencv, I get a frame from the camera containing vehicles, remove the background and track the vehicles, but before I do this I do some pre-processing and post-processing like removing noise with blur, all this runs in a single thread and it works great but here comes an issue, I now want to read number plates, for this i need a higher resolution frame otherwise for every frame I will not detect a single plate, but as soon as i increase my frame size I get a performance hit,my threads slows down to the point that my program no longer qualifies to be a real time system.
So I thought of adding more threads to my scene each to specialize on one task
here is a list of my tasks
//recieves fame from Camera
1. preprocess
//recieves a Frame from preprocess and removes the background
2. remove background
//recieves a Frame from backgroundremover and tracks the vehicles
3. postprocess
If I run the threads one by one am thinking it will still be slow instead I thought or running the threads simultenously but the issues it they use the same objects, declaring them volatile will mean threads waiting for the thread with lock to complete for it to use the object which will mean a slow system again so my question is how can I run these threads simultaneously without having to wait for others?
I have looked at a dozen of multithreading techniques in Java but finding it really hard to come up with a way of making this work.
So far I have looked at
1. Thread synchronization using the keyword volatile
2. Thread synchronization using the keyword synchronized
3. Multiple thread locks using a lock object
4. Using threadpools
5. Using the Countdown Latch
6. Wait and motify
7. Using Semaphores(which seemed like a good idea).
Here is the code I want to break down into those threads
public void VideoProcessor()
{
videProcessorThread = new Thread(new Runnable()
{
#Override
public void run()
{
try{
int i = 0;
while (isPlaying() && isMainScreenONOFF()) {
camera.read(frame);
//set default and max frame speed
camera.set(Videoio.CAP_PROP_FPS, 25);
//get frame speed just incase it did not set
fps = camera.get(Videoio.CAP_PROP_FPS);
//if(frame.height() > imgHeight || frame.width() > imgWidth)
Imgproc.resize(frame, frame, frameSize);
//check if to convert or not
if(getblackAndWhite())
Imgproc.cvtColor(frame, frame, Imgproc.COLOR_RGB2GRAY);
imag = frame.clone();
if(rOI){
//incase user adjusted the lines we try calculate there new sizes
adjustLinesPositionAndSize(xAxisSlider.getValue(), yAxisSlider.getValue());
//then we continue and draw the lines
if(!roadIdentified)
roadTypeIdentifier(getPointA1(), getPointA2());
}
viewClass.updateCarCounter(tracker.getCountAB(), tracker.getCountBA());
if (i == 0) {
// jFrame.setSize(FRAME_WIDTH, FRAME_HEIGHT);
diffFrame = new Mat(outbox.size(), CvType.CV_8UC1);
diffFrame = outbox.clone();
}
if (i == 1) {
diffFrame = new Mat(frame.size(), CvType.CV_8UC1);
removeBackground(frame, diffFrame, mBGSub, thresHold.getValue(), learningRate.getValue());
frame = diffFrame.clone();
array = detectionContours(diffFrame, maximumBlob.getValue(), minimumBlob.getValue());
Vector<VehicleTrack> detections = new Vector<>();
Iterator<Rect> it = array.iterator();
while (it.hasNext()) {
Rect obj = it.next();
int ObjectCenterX = (int) ((obj.tl().x + obj.br().x) / 2);
int ObjectCenterY = (int) ((obj.tl().y + obj.br().y) / 2);
//try counter
//add centroid and bounding rectangle
Point pt = new Point(ObjectCenterX, ObjectCenterY);
VehicleTrack track = new VehicleTrack(frame, pt, obj);
detections.add(track);
}
if (array.size() > 0) {
tracker.update(array, detections, imag);
Iterator<Rect> it3 = array.iterator();
while (it3.hasNext()) {
Rect obj = it3.next();
int ObjectCenterX = (int) ((obj.tl().x + obj.br().x) / 2);
int ObjectCenterY = (int) ((obj.tl().y + obj.br().y) / 2);
Point pt = null;
pt = new Point(ObjectCenterX, ObjectCenterY);
Imgproc.rectangle(imag, obj.br(), obj.tl(), new Scalar(0, 255, 0), 2);
Imgproc.circle(imag, pt, 1, new Scalar(0, 0, 255), 2);
//count and eleminate counted
tracker.removeCounted(tracker.tracks);
}
} else if (array.size() == 0) {
tracker.updateKalman(imag, detections);
}
}
i = 1;
//Convert Image and display to View
displayVideo();
}
//if error occur or video finishes
Image image = new Image("/assets/eyeMain.png");
viewClass.updateMainImageView(image);
}catch(Exception e)
{
e.printStackTrace();
System.out.println("Video Stopped Unexpectedly");
}
//thread is done
}
});videProcessorThread.start();
}
As no-one else has replied, I'll have a go.
You've already covered the main technical aspects in your questions (locking, synchronisation etc). Whichever way you look at it, there is no general solution to designing a multi-threaded system. If you have threads accessing the same objects you need to design your synchronisation and you can get threads blocking each other, slowing everything down.
The first thing to do is to do some performance profiling as there is no point making things run in parallel if they are not slowing things down.
That said, I think there are three approaches you could take in your case.
Have a single thread process each frame but have a pool of threads processing frames in parallel. If it takes a second to process a frame and you have 25fps you'd need at least 25 threads to keep up with the frame rate. You'd always be about a second behind real time but you should be able to keep up with the frame rate.
A typical way to implement this would be to put the incoming frames in a queue. You then have a pool of threads reading the latest frame from the queue and processing it. The downside of this design is that you can't guarantee in which order you would get the results of the processing so you might need to add some more logic around sorting the results.
The advantages are that:
There is very little contention, just around getting the frames off the queue, and that should be minimal
It is easy to tune and scale by adjusting the number of threads. It could even run on multiple machines, depending on how easy it is to move frames between machines
You avoid the overhead of creating a new thread each time as each thread processing one frame after another
It is easy to monitor as you can just look at the size of the queue
Error handling can be implemented easily, eg use ActiveMQ to re-queue a frame if a thread crashes.
Run parts of your algorithm in parallel. The way you've written it (pre-process, process, post-process), I don't see this is suitable as you can't do the post processing at the same time as the pre-processing. However, if you can express your algorithm in steps that can be run in parallel then it might work.
Try and run specific parts of your code in parallel. Looking at the code you posted the iterators are the obvious choice. Is there any reason not to run the iterator loops in parallel? If you can, experiment with the Java parallel streams to see if that bring any performance gains.
Personally I'd try option 1 first as its quick and simple.
I have approximately 40000 objects which might need to be repainted.
Most of them are not on the screen, so it seems that I could save a lot of work by doing the checks concurrently. But, my CPU never goes above 15% usage, so it seems that it is still only using one core. Have I implemented the threads correctly? If so, why aren't all my cores being used? And is there a better way which does utilize all my cores?
public void paintComponent(Graphics g)
{
super.paintComponent(g);
if (game.movables.size() > 10000)
{
final int size = game.drawables.size();
final Graphics gg = g;
Thread[] threads = new Thread[8];
for (int j = 0; j < 8; ++j)
{
final int n = j;
threads[j] = new Thread(new Runnable()
{
public void run()
{
Drawable drawMe;
int start = (size / 8) * n;
int end = (size / 8) * (n + 1);
if (n == 8) end = game.drawables.size(); // incase size
// % 8 != 0
for (int i = start; i < end; ++i)
{
drawMe = game.drawables.get(i);
if (drawMe.isOnScreen())
{
synchronized (gg)
{
drawMe.draw(gg);
}
}
}
}
});
threads[j].start();
}
try
{
for (int j = 0; j < 8; ++j)
threads[j].join();
}
catch (InterruptedException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
else
{
for (Drawable drawMe : game.drawables)
{
if (drawMe.isOnScreen())
{
drawMe.draw(g);
}
}
}
}
As has been pointed out, the synchronized (gg) is effectively serializing all the drawing, so you're probably going slower than single-threaded code due to thread creation and other overhead.
The main reason I'm writing however is that Swing, which this presumably is, is not thread safe. So the behavior of this program is not only likely to be bad, it's undefined.
Threading errors like this turn up as screwy behavior on some machines with some java runtime parameters and some graphics drivers. Been there. Done that. Not good.
JOGL will give you direct access to the GPU, the surest way to speed rendering.
To do this right, you might start by putting each drawMe in a (properly synchronized) list, then actually draw them in a loop after the joins are done. You can't speed the drawing (though if you've knocked out 99% of the drawMe's you've cut down the time needed dramatically), but if isOnScreen() is somewhat complicated, you'll get some real work out of your cores.
A ConcurrentLinkedQueue would save you the need to synchronize adds to the list.
The next step might be to use a blocking queue instead of a list, so the paint code could run in parallel with the visibility checks. With eight checks running, they should keep well ahead of the drawing. (But I think all the blocking queues either need synchronizing or do synching themselves. I'd skip this and stick with the CLQ and the first solution. Simpler and possibly faster.)
And (as Gene pointed out), everything Swing related starts on the EventQueue. Keep it there or life will get strange. Only your own code, not referencing the UI, should run in your threads.
Since you're already not drawing any objects that are off-screen, you're probably gaining very very little by doing what you're doing above.
I would also go as far as to say you're making it worse, but introducing synchronize which is slow and also introducing threads that cause context switches, which are expensive.
To improve performace you should perhaps look into using different drawing libraries, such as the Java2D drawing library, which is part of the JDK: http://java.sun.com/products/java-media/2D/index.jsp
I'm not sure how java will handle this, but other languages will blow up horribly and die if you reference something across scopes like you're doing with final int n (since it goes out of scope when the loop stops). Consider making it a field of the runnable object. Also, you're synchronizing on the graphics object while you're doing all of the real work. It's likely that you aren't getting any real performance increase from this. You might benefit from explicitly checking if the object is on the screen in parallel which is a read only operation, adding on-screen objects to a set or collection of some other sort, and then rendering sequentially.