We Keep Coding

OpenCV Android (java) character detection and font recognition

I'm working on android app, which determines which font is used on a text image. So I need to extract every character from image and don't know how to do it precisely. Furthermore, when I'm trying to process an image I have one result...but my classmate has different (for example, more or less noise). The problem with character detection is that:
1) it detects also some noise blobs on image and shows it in rectangles (I thought about detectMultiScale... but I have doubts about it, maybe there are easiest ways to detect characters)
2) it detects several contours of one character (for example inner and outer radius of letter "o")
And question for the future: I'm going to create a DB with images (for now just 3 fonts) of different letters of fonts and compare them with an image of letters from photo. Maybe someone could recommend a better way to do it.
So this is part of code with image processing(I'm still playing with values of blur, threshold and Canny... but there was no really positive result):
Imgproc.cvtColor(sImage, grayImage, Imgproc.COLOR_BGR2GRAY); //градации серого
Imgproc.GaussianBlur(grayImage,blurImage,new Size(5, 5),0); //размытие
Imgproc.adaptiveThreshold(blurImage, thresImage, 255, Imgproc.ADAPTIVE_THRESH_MEAN_C, Imgproc.THRESH_BINARY, 101, 39);
List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
Mat hierarchy = new Mat();
Imgproc.Canny(thresImage, binImage, 30, 10, 3, true); //контур
Imgproc.findContours(binImage, contours, hierarchy, Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE, new Point(0, 0));
hierarchy.release();
Imgproc.drawContours(binImage, contours, -1, new Scalar(255, 255, 255));//, 2, 8, hierarchy, 0, new Point());
MatOfPoint2f approxCurve = new MatOfPoint2f();
//For each contour found
for (int i = 0; i < contours.size(); i++) {
//Convert contours(i) from MatOfPoint to MatOfPoint2f
MatOfPoint2f contour2f = new MatOfPoint2f(contours.get(i).toArray());
//Processing on mMOP2f1 which is in type MatOfPoint2f
double approxDistance = Imgproc.arcLength(contour2f, true) * 0.02;
Imgproc.approxPolyDP(contour2f, approxCurve, approxDistance, true);
//Convert back to MatOfPoint
MatOfPoint points = new MatOfPoint(approxCurve.toArray());
// Get bounding rect of contour
Rect rect = Imgproc.boundingRect(points);
// draw enclosing rectangle (all same color, but you could use variable i to make them unique)
Imgproc.rectangle(binImage, new Point(rect.x, rect.y), new Point(rect.x + rect.width, rect.y + rect.height), new Scalar(255, 255, 255), 5);
}
And screen (not actually with processing values from code, just one with better results):
Original:
(unfortunately, I can't add more than 2 links to show more examples)
There were situations, when picture from this screen looked pretty good, but another pictures looked like with shapeless blobs.

Your code is fine, you just need to make a minor tweaks to get it work properly.
Firstly, the image size is very large, you can safely reduce it to 20% of current size without suffering a major loss in accuracy. Due to larger image size all the functions would perform slower.
You dont need to perform adaptive threshold before Canny, canny works perfectly on gray-scale images as well, You need to adjust the params as:
Canny(img, threshold1=170, threshold2=250)
which yields an image as:
[Optional] If you want to de-noise the image then you can try with morphological operations like erode and dilate.
Now you are ready to find the contours. The mistake in your code was using Imgproc.RETR_TREE flag you need to use Imgproc.RETR_EXTERNAL flag to get only the outer contours and not the nested inner contours.
At this step you may have some unwanted small contours, which can be filtered as:
// ** Below code if for reference purposes only, consult OpenCV docs for proper API methods
int character_area_lower_thresh = 10;
for (Contour c:contours) {
if (Imgproc.contourArea(c) > character_area_lower_thresh) {
// Desired contour, do what ever you want to do
Rect r = Imgproc.boundingRect(c);
}
}

OpenCV - Ellipse not showing at all

I wanna make an ellipse mask for cropping an image so only the contents inside of the ellipse will be shown.
Could you inspect my code?
public static Mat cropImage(Mat imageOrig, MatOfPoint contour){
Rect rect = Imgproc.boundingRect(contour);
MatOfPoint2f contour2f = new MatOfPoint2f(contour.toArray());
RotatedRect boundElps = Imgproc.fitEllipse(contour2f);
Mat out = imageOrig.submat(rect);
// the line function is working
Imgproc.line(out, new Point(0,0), new Point(out.width(), out.height()), new Scalar(0,0,255), 5);
// but not this one
Imgproc.ellipse(out, boundElps, new Scalar(255, 0, 0), 99);
return out;
}//cropImage
It seems like it's not working at all. Though you can see the line function I've done to test if it is working on the right image and I can see a line but there's no ellipse.
Here's a sample output of my cropImage function.
TIA

You're retrieving the ellipse coordinates in the imageOrig coordinates system, but you're showing it on the cropped out image.
If you want to show the ellipse on the crop, you need to translate the ellipse center to account for the translation introduced by the crop (top-left coordinates of rect), something like:
boundElps.center().x -= rect.x; boundElps.center().y -= rect.y;

You can try this out:
RotatedRect rRect = Imgproc.minAreaRect(contour2f);
Imgproc.ellipse(out, rRect , new Scalar(255, 0, 0), 3);

You should check for the minimum requirements for using the fitEllipse as shown in this post.
The function fitEllipse requires atleast 5 points.
Note: Although the reference I mention is for Python, I hope you can do the same for Java.
for cnt in contours:
area = cv2.contourArea(cnt)
# Probably this can help but not required
if area < 2000 or area > 4000:
continue
# This is the check I'm referring to
if len(cnt) < 5:
continue
ellipse = cv2.fitEllipse(cnt)
cv2.ellipse(roi, ellipse, (0, 255, 0), 2)
Hope it helps!

What does cvHaarDetectObjects() method do?

Please can some expert person explain me whether we can use the cvHaarDetectObjects() method to detect squares and get width and heights? I found a code that use this method for face-detection but I need to know whether I can use it for rectangle detection.
String src="src/squiredetection/MY.JPG";
IplImage grabbedImage = cvLoadImage(src);
IplImage grayImage = IplImage.create(grabbedImage.width(), grabbedImage.height(), IPL_DEPTH_8U, 1);
cvCvtColor(grabbedImage, grayImage, CV_BGR2GRAY);
CvSeq faces = cvHaarDetectObjects(grayImage, cascade, storage, 1.1, 3, 0);//*
for (int i = 0; i < faces.total(); i++) {
CvRect r = new CvRect(cvGetSeqElem(faces, i));
cvRectangle(grabbedImage, cvPoint(r.x(), r.y()), cvPoint(r.x()+r.width(), r.y()+r.height()), CvScalar.RED, 1, CV_AA, 0);
/* hatPoints[0].x = r.x-r.width/10; hatPoints[0].y = r.y-r.height/10;
hatPoints[1].x = r.x+r.width*11/10; hatPoints[1].y = r.y-r.height/10;
hatPoints[2].x = r.x+r.width/2; hatPoints[2].y = r.y-r.height/2;*/
// cvFillConvexPoly(grabbedImage, hatPoints, hatPoints.length, CvScalar.GREEN, CV_AA, 0);
}
when I use above method it throws following exception
OpenCV Error: Bad argument (Invalid classifier cascade) in unknown function, file C:\slave\WinInstallerMegaPack\src\opencv\modules\objdetect\src\haar.cpp, line 1036
Exception in thread "main" java.lang.RuntimeException: C:\slave\WinInstallerMegaPack\src\opencv\modules\objdetect\src\haar.cpp:1036: error: (-5) Invalid classifier cascade
at com.googlecode.javacv.cpp.opencv_objdetect.cvHaarDetectObjects(Native Method)
at com.googlecode.javacv.cpp.opencv_objdetect.cvHaarDetectObjects(opencv_objdetect.java:243)
at squiredetection.Test2.main(Test2.java:52 I have put * on this line)
Please be kind enough to give simple code example for that.

cvHaarDetectObjects() is used for detecting objects or shapes not only for faces, it depends on HaarCascade classifier.
If you pass face haarcascade xml then it will return an array of faces or also can use eye, nose, etc HaarCascade XML file. You can make also custom haarcascade xml by creating your own positive and negative samples using opencv_traincascade.exe
CvSeq faces = cvHaarDetectObjects(grayImage, classifier, storage,
1.1, 3, CV_HAAR_DO_CANNY_PRUNING);
for (int i = 0; i < faces.total(); i++) {
// its ok
}
detail on opencv doc
for rectangle detection :
there is an example for rectangle detection in OpenCV, they use it to detect
the squares in a chessboard. Have a look to squares.c in
..\OpenCV\samples\c\ directory.
see this chessboard detection sample in opencv
Invalid classifier cascade in unknown function error means the classifier you passed is not correctly formatted or something is missing. Check if your classifier xml file is valid.

cvHaarDetectObjects returns multiple faces detected in an image. You have to declare an array of CvSeq to store the result, not just a single CvSeq.
// There can be more than one face in an image.
// So create a growable sequence of faces.
// Detect the objects and store them in the sequence
CvSeq* faces = cvHaarDetectObjects( img, cascade, storage,
1.1, 2, CV_HAAR_DO_CANNY_PRUNING,
cvSize(40, 40) );
The code above was extracted from this site:
http://opencv.willowgarage.com/wiki/FaceDetection

Equivalent of OpenCV statement in Java using JavaCV

I want to know how to construct the following C++ statement in OpenCV using JavaCV:
float* p = (float*)cvGetSeqElem(circles, i);
int radius = cvRound(p[2]);
To get the Radius of a circle detected using cvHoughCircles(). Obviously Java doesn't use pointer so I have no idea how to do this in Java. The code I have so far so you can see it context:
lines = cvHoughCircles(frame2, storage, CV_HOUGH_GRADIENT, 1, 50, 300, 60, 10, 600);
for (int i = 0; i < lines.total(); i++) {
//Would like the code to go here
CvPoint2D32f point = new CvPoint2D32f(cvGetSeqElem(lines, i));
cvCircle(src, cvPoint((int)point.x(), (int)point.y()), 3, CvScalar.WHITE, -1, 8, 0);
Point p = new Point((int)point.x(), (int)point.y());
points.add(p);
}

JavaCPP maps C/C++ arrays/pointers to Pointer objects, so we can access it in the same way as in C/C++, i.e.:
FloatPointer p = new FloatPointer(cvGetSeqElem(circles, i));
int radius = Math.round(p.get(2));

Using warpPerspective() on a sequence of points given by HoughCircles(), OpenCV

I'm trying to detect the positions of billiards balls on a table from an image taken at a perspective angle. I'm using the getPerspectiveTransform() method to find the transformation matrix and I want to apply that to only the circles I detect using HoughCircles. I'm trying to go from a rather large trapezoidal shape to a smaller rectangular shape. I don't want to do the transformation on the image first and then find the HoughCircles because the image gets too warped for houghcircles to provide useful results.
Here's my code:
CvMat mmat = cvCreateMat(3,3,CV_32FC1);
double srcX1 = 462;
double srcX2 = 978;
double srcX3 = 1440;
double srcX4 = 0;
double srcY = 241;
double srcHeight = 772;
double dstX = 56.8;
double dstY = 33.5;
double dstWidth = 262.4;
double dstHeight = 447.3;
CvSeq seq = cvHoughCircles(newGray, circles, CV_HOUGH_GRADIENT, 2.1d, (double)newGray.height()/40, 85d, 65d, 5, 50);
JavaCV.getPerspectiveTransform(new double[]{srcX1, srcY, srcX2,srcY, srcX3, srcHeight, srcX4, srcHeight},
new double[]{dstX, dstY, dstWidth, dstY, dstWidth, dstHeight, dstX, dstHeight}, mmat);
cvWarpPerspective(seq, seq, mmat);
for(int j=0; j<seq.total(); j++){
CvPoint3D32f point = new CvPoint3D32f(cvGetSeqElem(seq, j));
float xyr[] = {point.x(),point.y(),point.z()};
CvPoint center = new CvPoint(Math.round(xyr[0]), Math.round(xyr[1]));
int radius = Math.round(xyr[2]);
cvCircle(gray, center, 3, CvScalar.GREEN, -1, 8, 0);
cvCircle(gray, center, radius, CvScalar.BLUE, 3, 8, 0);
}
The problem is I get this error on the warpPerspective() method:
error: (-215) seq->total > 0 && CV_ELEM_SIZE(seq->flags) == seq->elem_size in function cv::Mat cv::cvarrToMat(const CvArr*, bool, bool, int)
Also I guess it's worth mentioning that I'm using JavaCV, in case the method calls look a bit different than what you're used to. Thanks for any help.

Answer:
the problem with what you want to do (besides the obvious, opencv wont let you) is that the radius cant really be warped correctly. AFAIK the xy coordinates are pretty easy to calculate x'=((m00x+m01y+m02)/(m20x+m21y+m22)) y'=((m10x+m11y+m12)/(m20x+m21y_m22)) when m is the transformation matrix. the radius you can hack by transforming all the points of the original circle and then find the max distance between x'y' and those points (atleast if the radius in the warped image is expected to cover all those points)
btw, mIJx = m(i,j)*x (just to clarify)
End Answer.
Everything i write is according to the c++ version, i've never used JavaCV but from what i could see its just a wrapper that calls the native c++ lib.
CvSeq is a sequance data structure that behaves like a linked list.
the assert your application crushes at is
CV_Assert(seq->total > 0 && CV_ELEM_SIZE(seq->flags) == seq->elem_size);
which means that either your seq instance is empty (total is the number of elements in the sequence) or somehow the inner seq flags are corrupted.
I'd recommend that you'd check the total member of your CvSeq, and the cvHoughCircles call.
all of this occurs before the actual implementation of cvWarpPerspective (its the first line in the implementation, that only converts your CvSeq to cv::Mat).. so its not the warping but what you're doing before that.
anyway, to understand whats wrong with cvHoughCircles we'll need more info about the creation of newGray and circles.
here is an example i've found on the javaCV page (Link)
IplImage gray = cvCreateImage( cvSize( img.width, img.height ), IPL_DEPTH_8U, 1 );
cvCvtColor( img, gray, CV_RGB2GRAY );
// smooth it, otherwise a lot of false circles may be detected
cvSmooth(gray,gray,CV_GAUSSIAN,9,9,2,2);
CvMemStorage circles = CvMemStorage.create();
CvSeq seq = cvHoughCircles(gray, circles.getPointer(), CV_HOUGH_GRADIENT,
2, img.height/4, 100, 100, 0, 0);
for(int i=0; i<seq.total; i++){
float xyr[] = cvGetSeqElem(seq,i).getFloatArray(0, 3);
CvPoint center = new CvPoint(Math.round(xyr[0]), Math.round(xyr[1]));
int radius = Math.round(xyr[2]);
cvCircle(img, center.byValue(), 3, CvScalar.GREEN, -1, 8, 0);
cvCircle(img, center.byValue(), radius, CvScalar.BLUE, 3, 8, 0);
}
from what i've seen in the implementation of cvHoughCircles, the answer is saved in the circles buff and at the end they create from it the CvSeq to return, so if you've allocated the circles buff wrong, it wont work.
EDIT:
as you can see, the CvSeq instance in case of the return from cvHoughCircles is a list of point-values, that is probably why the assertion failed. you cannot convert this CvSeq into a cv::Mat.. because its just not a cv::Mat. to get only the circles returned from cvHoughCircles in an cv::Mat instance, you'll need to create a new cv::Mat instance and than draw onto it all the circles in the CvSeq - as seen in the provided example above.
than the warping will work (you'll have a cv::Mat instance, and that is what the function expect - a cv::Mat as the only element in the CvSeq)
END EDIT
here is the c++ reference for CvSeq
and if you want to fiddle with the source code than
cvarrToMat is in matrix.cpp
CV_ELEM_SIZE is in types_c.h
cvWarpPerspective is in imgwarp.cpp
cvHoughCircles is in hough.cpp
I hope that will help.
BTW, your next error will probably be:
cv::warpPerspective in the C++ OpencCv asserts that dst.data != src.data
thus
cvWarpPerspective(seq, seq, mmat);
wont work cause your source mat and destination mat referencing the same data.
Not all the functions in OpenCV (and image processing in general) work in-situ (because there is no in-situ algorithm or because its slower then the other version eg. transpose of an n*n mat will work in-situ, but n*m where n!=m will be harder to do in-situ and might be slower)
you cant assume the using the src matrix as the dst will work.