I am Working on Sign Translator Application For Handy People. In My application User Will Give one sign image from camera or gallery given image will be compare with database images and show the result With Alphabetic Sign.
but my problem is i am not getting good similarity between two images Some Time result is accurate some time not.
Please Refer me Some Idea Or source Code.
Thanks in advance.
Scalar lowerThreshold = new Scalar(0, 48, 80); // Blue color – lower hsv values
Scalar upperThreshold = new Scalar(20, 255, 255); // Blue color – higher hsv values
FeatureDetector detector = FeatureDetector.create(FeatureDetector.PYRAMID_FAST);
DescriptorExtractor extractor = DescriptorExtractor.create(DescriptorExtractor.ORB);
//orb orb bruteforce with filter method
DescriptorMatcher matcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE_HAMMING);
//crash on surf flanbased
Mat img1 = new Mat();
Mat img2 = new Mat();
Utils.bitmapToMat(defaultImage,img1);
Utils.bitmapToMat(databaseImage,img2);
Mat descriptors1 = new Mat();
MatOfKeyPoint keypoints1 = new MatOfKeyPoint();
detector.detect(img1, keypoints1);
extractor.compute(img1, keypoints1, descriptors1);
//second image
Mat descriptors2 = new Mat();
MatOfKeyPoint keypoints2 = new MatOfKeyPoint();
detector.detect(img2, keypoints2);
extractor.compute(img2, keypoints2, descriptors2);
//matcher image descriptors
MatOfDMatch matches = new MatOfDMatch();
matcher.match(descriptors1,descriptors2,matches);
//Filter matches by distance
MatOfDMatch filtered = filterMatchesByDistance(matches);
int total = (int) matches.size().height;
int Match= (int) filtered.size().height;
Log.d("LOG", "total:" + total + " Match:"+Match);
int percent = (int)((Match * 100.0f) / total);
if(percent>max){
max=percent;
maximumPercentage.setMaximum(percent);
maximumPercentage.setImageId(id);
imageId=id;
Log.d("Maximum Percentage: ",String.valueOf(max)+"%");
Log.d("MaxId: ",String.valueOf(imageId));
}
id++;
Log.d("matchingOImages: ",String.valueOf(percent)+"%");
filter matching result method
List<DMatch> matches_original = matches.toList();
List<DMatch> matches_filtered = new ArrayList<DMatch>();
int DIST_LIMIT = 30;
// Check all the matches distance and if it passes add to list of filtered matches
Log.d("DISTFILTER", "ORG SIZE:" + matches_original.size() + "");
for (int i = 0; i < matches_original.size(); i++) {
DMatch d = matches_original.get(i);
if (Math.abs(d.distance) <= DIST_LIMIT) {
matches_filtered.add(d);
}
}
Log.d("DISTFILTER", "FIL SIZE:" + matches_filtered.size() + "");
MatOfDMatch mat = new MatOfDMatch();
mat.fromList(matches_filtered);
return mat;
Ok well i think you just entered the modern age of neural networks.
As it can be overwhelming how this stuff works, and often takes years of study, there are some shortcuts to get things done.
For the quickest result I think you might start here:
( Assuming you rather dont want to dive that deep into the innerworkings of a neural net, but rather would use existing software, or services ) https://cloud.google.com/automl/
Related
please help me with this as I am new to image processing.
I have the following images, there is glitterings/powder on the surface of the hand. How do I go about detecting these things on the hand?
I have tried with detecting by getting 70% of the max Intensity in the image. However, only image one works the rest does not. Could anyone how to suggest any methods that I can use to perform the detection please.(any available codes to try with will be good) Thank you?
Input Image 1: The only image that works with the above code
Input Image 2
Input Image 3
Desired Outcome
//convert from bitmap to mat
Mat mat = new Mat(bitmap1.getWidth(), bitmap1.getHeight(), CvType.CV_8UC3);
Utils.bitmapToMat(bitmap1, mat);
Mat grayMat = new Mat();
Imgproc.cvtColor(mat, grayMat, Imgproc.COLOR_BGR2GRAY);
//Log.d("TAGG","intensity" + mat.dump());
int rows = mat.rows();
int cols = mat.cols();
double maxIntensity = Core.minMaxLoc(grayMat).maxVal;
double minIntensity = 0.7 * maxIntensity;
Log.d("TAGG", "intensity" + maxIntensity);
Mat thresholdMat = new Mat();
Imgproc.threshold(grayMat, thresholdMat, minIntensity, maxIntensity, Imgproc.THRESH_BINARY_INV);
Bitmap outputBitmap = Bitmap.createBitmap(thresholdMat.cols(), thresholdMat.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(thresholdMat, outputBitmap);
I want to detect octagonal stop sign like the following image with contours but i can't figure out how it is done
Stop Sign:
I already manage to detect triangles
Mat ROI = new Mat();
Mat bgrClone = bgr.clone();
MatOfPoint approxContour = new MatOfPoint();
MatOfPoint2f approxContour2f = new MatOfPoint2f();
List<MatOfPoint> contourDraw = new ArrayList<MatOfPoint>();
for(int i = 0; i < contourList.size(); i++) {
MatOfPoint2f contour2f = new MatOfPoint2f(contourList.get(i).toArray());
double approxDistance = Imgproc.arcLength(contour2f, true) * 0.02;//0.225
Imgproc.approxPolyDP(contour2f, approxContour2f, approxDistance, true);
approxContour2f.convertTo(approxContour, CvType.CV_32S);
if (approxContour.size().height == 3 && (Imgproc.contourArea(contour2f) > 3000) ) { //&& (Imgproc.contourArea(contour2f) > 5000)
contourDraw.add(approxContour);
Imgproc.drawContours(bgr, contourDraw, -1, new Scalar(0,255,0), 1);
Rect cord = Imgproc.boundingRect(approxContour);
Core.rectangle(bgr, new Point(cord.x, cord.y), new Point(cord.x+cord.width, cord.y+cord.height),new Scalar(0,255,0), 1);
ROI = bgrClone.submat(cord.y, cord.y+cord.height, cord.x, cord.x+cord.width);
showResult(ROI);
}
}
I am using Java but any in any language would be appreciated.
You can perform shape detection using contours with OpenCV. I implemented the following in python.
Steps:
Since you are dealing with traffic signs, most of them would be red in color. In the RGB color space the amount of red is the highest in the red channel. But since white color also comprises of red I did not use it. I rather used the blue channel where red content is minimal.
Blue channel image:
Next I applied Otsu threshold to the image above and inverted it.
Otsu threshold and Inversion:
Finally I found contours and marked those contours having roughly 8 arcs around it.
Final Result:
Code:
import numpy as np
import cv2
import os
path = r'C:\Desktop\Stack\contour\shape_detection'
filename = 'stop.png'
img = cv2.imread(os.path.join(path, filename), 1)
cv2.imshow('img1',img[:,:,0])
ret,thresh1 = cv2.threshold(img[:,:,0], 0, 255,cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)
cv2.imshow('thresh1', thresh1)
_, contours, hierarchy = cv2.findContours(thresh1, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
for cnt in contours:
approx = cv2.approxPolyDP(cnt,0.01*cv2.arcLength(cnt,True),True)
print len(approx)
if len(approx)==8:
print "octagon"
cv2.drawContours(img, [cnt], 0, (0, 255, 0), 6)
cv2.imshow('sign', img)
cv2.waitKey(0)
cv2.destroyAllWindows()
The below Algorithm returns image comparison percentage using openCv Java.
private double compare(Mat hist1, String f2) {
double compare = 0;
ArrayList<Mat> bgr_planes2 = new ArrayList<Mat>();
boolean accumulate = false;
MatOfInt histSize = new MatOfInt(180);
MatOfInt channels = new MatOfInt(0);
MatOfFloat histRanges = new MatOfFloat(0f, 180f);
// Mat img2 = Imgcodecs.imread(f2);
// img2 = resize(img2);
Mat img2 = ContourUtils.findContour(f2, false);
Imgproc.cvtColor(img2, img2, Imgproc.COLOR_RGBA2GRAY);
img2.convertTo(img2, CvType.CV_32F);
Mat hist2 = new Mat();
Core.split(img2, bgr_planes2);
Imgproc.calcHist(bgr_planes2, channels, new Mat(), hist2, histSize,
histRanges, accumulate);
Core.normalize(hist2, hist2, 0, hist2.rows(), Core.NORM_MINMAX, -1,
new Mat());
// img2.convertTo(img2, CvType.CV_32F);
hist2.convertTo(hist2, CvType.CV_32F);
compare = Imgproc.compareHist(hist1, hist2, Imgproc.CV_COMP_CORREL);
img2.release();
hist2.release();
channels.release();
histRanges.release();
histSize.release();
for (Mat m : bgr_planes2) {
if (m != null) {
m.release();
}
}
return compare;
}
Same way i want to find the ranking instead of percentage that how well the images are matched. Please suggest me an idea for ranking algorithm. Thanks.
I understand that your algorithm is comparing an image to a reference image. If you want a ranking of the best math, apply the function to your collection of input image, then sorting the results (descending order) provide you the ranking.
Store the result as key and the list of names of input image (expecting many inputs could have the same result value) as value of a Map.
For example:
Map<double, List<String>> ranking
I am working on a real time text detection and recognition with OpenCV4Android. Recognition part is totally completed. However, I have to ask question about text detection. I' m using the MSER FeatureDetector for detection text.
This is the real time and calling the method part:
public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame) {
carrierMat = inputFrame.gray();
carrierMat = General.MSER(carrierMat);
return carrierMat;
}
And this is the basic MSER implementation:
private static FeatureDetector fd = FeatureDetector.create(FeatureDetector.MSER);
private static MatOfKeyPoint mokp = new MatOfKeyPoint();
private static Mat edges = new Mat();
public static Mat MSER(Mat mat) {
//for mask
Imgproc.Canny(mat, edges, 400, 450);
fd.detect(mat, mokp, edges);
//for drawing keypoints
Features2d.drawKeypoints(mat, mokp, mat);
return mat;
}
It works fine for finding text with edges mask.
I would like to draw a rectangles for clusters like this:
or this:
You can assume that I have the right points.
As you can see, fd.detect() method is returning a MatOfKeyPoint. Hence I' ve tried this method for drawing rectangle:
public static Mat MSER_(Mat mat) {
fd.detect(mat, mokp);
KeyPoint[] refKp = mokp.toArray();
Point[] refPts = new Point[refKp.length];
for (int i = 0; i < refKp.length; i++) {
refPts[i] = refKp[i].pt;
}
MatOfPoint2f refMatPt = new MatOfPoint2f(refPts);
MatOfPoint2f approxCurve = new MatOfPoint2f();
//Processing on mMOP2f1 which is in type MatOfPoint2f
double approxDistance = Imgproc.arcLength(refMatPt, true) * 0.02;
Imgproc.approxPolyDP(refMatPt, approxCurve, approxDistance, true);
//Convert back to MatOfPoint
MatOfPoint points = new MatOfPoint(approxCurve.toArray());
// Get bounding rect
Rect rect = Imgproc.boundingRect(points);
// draw enclosing rectangle (all same color, but you could use variable i to make them unique)
Imgproc.rectangle(mat, new Point(rect.x, rect.y), new Point(rect.x + rect.width, rect.y + rect.height), Detect_Color_, 5);
//Features2d.drawKeypoints(mat, mokp, mat);
return mat;
}
But when I was trying to Imgproc.arcLength() method, it suddenly stopped. I gave a random approxDistance value for Imgproc.approxPolyDP() method like 0.1, it doesn' t work really efficiently.
So how can I draw rectangle for detected text?
I tested your code and had exactly the same problem.
For now I still can't find the problem within.
But I found a project using both "MSER" and "Morphological".
you can find it here .
The project have very simple structure and the author put the
text detection in "onCameraFrame" method just like you.
I implemented the method from that project and it worked,
but the result was still not very good.
If you seek better text detection tool, here's two of them.
Stroke Width Transform(SWT):
A whole new method for finding text area. It's fast and efficient. however it is only available in c++ or python. you can find some example here.
Class-specific Extremal Regions using class ERFilter:An advanced version of the MSER. Unfortunately, it is only available in OpenCV 3.0.0-dev. You can't use it in current version of OpenCV4Android. The document is here.
To be honest I am new in this area(2 months), but I hope these information can help you finish your project.
(update:2015/9/13)
I've translated a c++ method from a post.
It works far better than the first github project I mentioned.
Here is the code:
public void apply(Mat src, Mat dst) {
if (dst != src) {
src.copyTo(dst);
}
Mat img_gray,img_sobel, img_threshold, element;
img_gray=new Mat();
Imgproc.cvtColor(src, img_gray, Imgproc.COLOR_RGB2GRAY);
img_sobel=new Mat();
Imgproc.Sobel(img_gray, img_sobel, CvType.CV_8U, 1, 0, 3, 1, 0,Core.BORDER_DEFAULT);
img_threshold=new Mat();
Imgproc.threshold(img_sobel, img_threshold, 0, 255, Imgproc.THRESH_OTSU+Imgproc.THRESH_BINARY);
element=new Mat();
element = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(17, 3) );
Imgproc.morphologyEx(img_threshold, img_threshold, Imgproc.MORPH_CLOSE, element);
//Does the trick
List<MatOfPoint> contours=new ArrayList<MatOfPoint>();
Mat hierarchy = new Mat();
Imgproc.findContours(img_threshold, contours, hierarchy, 0, 1);
List<MatOfPoint> contours_poly=new ArrayList<MatOfPoint>(contours.size());
contours_poly.addAll(contours);
MatOfPoint2f mMOP2f1,mMOP2f2;
mMOP2f1=new MatOfPoint2f();
mMOP2f2=new MatOfPoint2f();
for( int i = 0; i < contours.size(); i++ )
if (contours.get(i).toList().size()>100)
{
contours.get(i).convertTo(mMOP2f1, CvType.CV_32FC2);
Imgproc.approxPolyDP(mMOP2f1,mMOP2f2, 3, true );
mMOP2f2.convertTo(contours_poly.get(i), CvType.CV_32S);
Rect appRect=Imgproc.boundingRect(contours_poly.get(i));
if (appRect.width>appRect.height)
{
Imgproc.rectangle(dst, new Point(appRect.x,appRect.y) ,new Point(appRect.x+appRect.width,appRect.y+appRect.height), new Scalar(255,0,0));
}
}
}
I'm implementing using Java the OpenCV tutorial for finding an object in a scene using homography http://docs.opencv.org/doc/tutorials/features2d/feature_homography/feature_homography.html#feature-homography
Below is my implementation, where img1 is the scene and img2 is the object
FeatureDetector detector = FeatureDetector.create(FeatureDetector.ORB);
DescriptorExtractor descriptor = DescriptorExtractor.create(DescriptorExtractor.ORB);
DescriptorMatcher matcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE);
//set up img1 (scene)
Mat descriptors1 = new Mat();
MatOfKeyPoint keypoints1 = new MatOfKeyPoint();
//calculate descriptor for img1
detector.detect(img1, keypoints1);
descriptor.compute(img1, keypoints1, descriptors1);
//set up img2 (template)
Mat descriptors2 = new Mat();
MatOfKeyPoint keypoints2 = new MatOfKeyPoint();
//calculate descriptor for img2
detector.detect(img2, keypoints2);
descriptor.compute(img2, keypoints2, descriptors2);
//match 2 images' descriptors
MatOfDMatch matches = new MatOfDMatch();
matcher.match(descriptors1, descriptors2,matches);
//calculate max and min distances between keypoints
double max_dist=0;double min_dist=99;
List<DMatch> matchesList = matches.toList();
for(int i=0;i<descriptors1.rows();i++)
{
double dist = matchesList.get(i).distance;
if (dist<min_dist) min_dist = dist;
if (dist>max_dist) max_dist = dist;
}
//set up good matches, add matches if close enough
LinkedList<DMatch> good_matches = new LinkedList<DMatch>();
MatOfDMatch gm = new MatOfDMatch();
for (int i=0;i<descriptors2.rows();i++)
{
if(matchesList.get(i).distance<3*min_dist)
{
good_matches.addLast(matchesList.get(i));
}
}
gm.fromList(good_matches);
//put keypoints mats into lists
List<KeyPoint> keypoints1_List = keypoints1.toList();
List<KeyPoint> keypoints2_List = keypoints2.toList();
//put keypoints into point2f mats so calib3d can use them to find homography
LinkedList<Point> objList = new LinkedList<Point>();
LinkedList<Point> sceneList = new LinkedList<Point>();
for(int i=0;i<good_matches.size();i++)
{
objList.addLast(keypoints2_List.get(good_matches.get(i).queryIdx).pt);
sceneList.addLast(keypoints1_List.get(good_matches.get(i).trainIdx).pt);
}
MatOfPoint2f obj = new MatOfPoint2f();
MatOfPoint2f scene = new MatOfPoint2f();
obj.fromList(objList);
scene.fromList(sceneList);
//output image
Mat outputImg = new Mat();
MatOfByte drawnMatches = new MatOfByte();
Features2d.drawMatches(img1, keypoints1, img2, keypoints2, gm, outputImg, Scalar.all(-1), Scalar.all(-1), drawnMatches,Features2d.NOT_DRAW_SINGLE_POINTS);
//run homography on object and scene points
Mat H = Calib3d.findHomography(obj, scene,Calib3d.RANSAC, 5);
Mat tmp_corners = new Mat(4,1,CvType.CV_32FC2);
Mat scene_corners = new Mat(4,1,CvType.CV_32FC2);
//get corners from object
tmp_corners.put(0, 0, new double[] {0,0});
tmp_corners.put(1, 0, new double[] {img2.cols(),0});
tmp_corners.put(2, 0, new double[] {img2.cols(),img2.rows()});
tmp_corners.put(3, 0, new double[] {0,img2.rows()});
Core.perspectiveTransform(tmp_corners,scene_corners, H);
Core.line(outputImg, new Point(scene_corners.get(0,0)), new Point(scene_corners.get(1,0)), new Scalar(0, 255, 0),4);
Core.line(outputImg, new Point(scene_corners.get(1,0)), new Point(scene_corners.get(2,0)), new Scalar(0, 255, 0),4);
Core.line(outputImg, new Point(scene_corners.get(2,0)), new Point(scene_corners.get(3,0)), new Scalar(0, 255, 0),4);
Core.line(outputImg, new Point(scene_corners.get(3,0)), new Point(scene_corners.get(0,0)), new Scalar(0, 255, 0),4);
The program is able to calculate and display feature points from both images. However, the scene_corners returned are 4 points in a close cluster (small green blob)
where they are supposed to represent the 4 corners of the perspective projection of the object onto the scene. I checked double checked to make sure my program is as close to the c++ implementation as possible. What might be causing this?
I checked the homography matrix and it seems the corner coordinates are skewed by 2 very big results from the matrix. Is the homography matrix incorrectly calculated?
I'd appreciate any input, thanks.
Update:
I played about with the filter threshold for good matches and found that 2.75*min_dist seems to work well with this set of images. I can now get good matches with zero outliers. However, the bounding box is still wrong. http://i.imgur.com/fuXeOqL.png
How do I know what value of threshold to use for best matches and how does the homography relate to them? Why was 3*min_dist used in the example?
I managed to solve the problem and use homography correctly while investigating index out of bounds errors. It turns out when I added my good matches to my object and scene lists, I swapped round the query and train indices
objList.addLast(keypoints2_List.get(good_matches.get(i).queryIdx).pt);
sceneList.addLast(keypoints1_List.get(good_matches.get(i).trainIdx).pt);
According to this question OpenCV drawMatches -- queryIdx and trainIdx , since I called
matcher.match(descriptors1, descriptors2,matches);
with descriptor1 first then descriptor2, the correct indices should be:
objList.addLast(keypoints2_List.get(good_matches.get(i).trainIdx).pt);
sceneList.addLast(keypoints1_List.get(good_matches.get(i).queryIdx).pt);
where queryIdx refers to keypoints1_List and trainIdx refers to keypoints2_List.
Here is an example result:
http://i.imgur.com/LZNBjY2.png
Currently I'm also implementing a 2D homography in java and I also found the OpenCV tutorial then your question.
I don't think it'll enhance your results but in the OpenCV tutorial when they compute the min and max distance, they loop with descriptors_object.rows and in your code you do with descriptors1.rows() which is the scene descriptor and not the object descriptor.
Edit: Just also noticed the same with the matcher. For you:
img1/descriptor1 -> Scene
img2/descriptor2 -> the object to find
In the tutorial:
matcher.match( descriptors_object, descriptors_scene, matches );
But in your code:
matcher.match(descriptors1, descriptors2,matches);
And Javadoc:
void org.opencv.features2d.DescriptorMatcher.match(Mat queryDescriptors, Mat trainDescriptors, MatOfDMatch matches)