I'm currently trying to improve the recognition rate of GoogleCloud Vision, so I am building a preprocessing pipeline.
I currently can create a mask which overlays the characters in the image, but as you can see in the examples below, it also shows the lines. Now since those lines can cross through characters, I'd like to remove them from the mask without destroying the characters, if possible.
Current steps:
Line detection:
InputImage -> Grayscale -> Blackhat -> GaussianBlur -> Threshhold(OTSU) -> HoughLinesP
Mask generation: InputImage -> Grayscale -> Blackhat -> GaussianBlur -> Threshhold(OTSU)-> ConnectedComponents
ImageExamples:(Due to privacy protection, sharing a full Image is not possible)
The images show the original image, the mask and the lines recognized.
The following code is used to generate the mask and find the lines
Mat picture = Imgcodecs.imread(path);
Imgproc.cvtColor(picture, picture, Imgproc.COLOR_BGR2GRAY);
Imgcodecs.imwrite("/home/meik/Pictures/asdfGray.png", picture);
Mat blackhatElement = Imgproc.getStructuringElement(Imgproc.CV_SHAPE_RECT, new Size(7, 7));
Imgproc.morphologyEx(picture, picture, Imgproc.MORPH_BLACKHAT, blackhatElement);
Imgproc.GaussianBlur(picture, picture, new Size(5, 3), 0);
Imgproc.threshold(picture, picture, 0, 255, Imgproc.THRESH_BINARY | Imgproc.THRESH_OTSU);
/**
* Line Detection with Canny and HoughLines(P)
*/
Mat lines = new Mat();
Mat linesResult = Mat.zeros(picture.rows(),picture.cols(), CvType.CV_8UC1);
Imgproc.HoughLinesP(picture, lines,1, Math.PI/180,100, 20, 0);
System.out.println("lines rows:" + lines.rows());
for (int x = 0; x < lines.rows(); x++) {
double[] l = lines.get(x, 0);
Imgproc.line(linesResult, new Point(l[0], l[1]), new Point(l[2], l[3]), new Scalar(255, 255, 255), 1, Imgproc.LINE_8, 0);
}
/**End of line detection*/
Mat kernel = Imgproc.getStructuringElement(Imgproc.CV_SHAPE_CROSS, new Size(3,3));
Imgproc.dilate(linesResult,linesResult,kernel);
Core.bitwise_not(linesResult,linesResult);
I've found this paper talking about the problem, but am struggling to understand their methodology.
How do I proceed from here on to remove lines without destroying the characters?
I dont really think you need to refer paper to do this.
Just use color info or hough line to find out a straightline which is really long
use that info to create a mask image.
Then use the opencv inpaint to remove it.
https://docs.opencv.org/2.4/modules/photo/doc/inpainting.html
e.g. what you want is similar to the bottom image. It ask to remove the traffic light poles. and you want the writing guideline to be removed. essentially, its the same thing
How about some simple image preprocessing?
For example using a threshold to only maintain a certain color range (instead of directly converting the image to grayscale).
Something like this is integrated in GIMP, see
https://docs.gimp.org/2.8/en/gimp-tool-threshold.html
You probably want to experiment with various thresholds.
Related
At the first contact with Java OpenCV (3.3.1, windows 8 x64) I'm trying to join two different size images with ROI dynamically. Here a bit of my code:
Mat _mat = Utils.imageFileToMat(new File("angelina_jolie.jpg")); //Angelina's face
Mat grayMat = new Mat();
Imgproc.cvtColor(_mat, grayMat, Imgproc.COLOR_BGR2GRAY);
Rect rect = new Rect(new Point(168, 104), new Point(254, 190)); //Angelina's eye ROI
Mat truncated = _mat.submat(rect); //Angelina's eye mat
Mat merge = _mat.clone();
truncated.copyTo(merge);
//show _mat
//show truncated
//show merge
What I want to see is Angelina Jolie with her eye on grayscale.
What I see is assertions or the truncated image only (just the eye).
I tried with copyTo(mat, mask), setOf, and a lot of things but always get a new assertion.
Should I change the size of truncated to the size of mat to match sizes? how can I do that programmatically?
Mat::copyTo documentation:
The method copies the matrix data to another matrix. Before copying
the data, the method invokes :
m.create(this->size(),this->type());
so that the destination matrix is reallocated
if needed. While m.copyTo(m); works flawlessly, the function does not
handle the case of a partial overlap between the source and the
destination matrices.
When the operation mask is specified, if the Mat::create call shown
above reallocates the matrix, the newly allocated matrix is
initialized with all zeros before copying the data.
#param m
Destination matrix. If it does not have a proper size or type before
the operation, it is reallocated.
Since you're your src and dst images don't have the same size and channels, the destination image is reallocated and initialized with zeros. To avoid that make sure both images have same dimensions and number of channels.
Imgproc.cvtColor(grayMat, grayMat, Imgproc.COLOR_GRAY2BGR);
Now create a mask:
Mat mask = new Mat(_mat.size(), CvType.CV_8UC1, new Scalar(0));
Imgproc.rectangle(mask, new Point(168, 104), new Point(254, 190),new Scalar(255));
// copy gray to _mat based on mask
Mat merge = _mat.clone();
grayMat.copyTo(merge,mask);
I'm working on android app, which determines which font is used on a text image. So I need to extract every character from image and don't know how to do it precisely. Furthermore, when I'm trying to process an image I have one result...but my classmate has different (for example, more or less noise). The problem with character detection is that:
1) it detects also some noise blobs on image and shows it in rectangles (I thought about detectMultiScale... but I have doubts about it, maybe there are easiest ways to detect characters)
2) it detects several contours of one character (for example inner and outer radius of letter "o")
And question for the future: I'm going to create a DB with images (for now just 3 fonts) of different letters of fonts and compare them with an image of letters from photo. Maybe someone could recommend a better way to do it.
So this is part of code with image processing(I'm still playing with values of blur, threshold and Canny... but there was no really positive result):
Imgproc.cvtColor(sImage, grayImage, Imgproc.COLOR_BGR2GRAY); //градации серого
Imgproc.GaussianBlur(grayImage,blurImage,new Size(5, 5),0); //размытие
Imgproc.adaptiveThreshold(blurImage, thresImage, 255, Imgproc.ADAPTIVE_THRESH_MEAN_C, Imgproc.THRESH_BINARY, 101, 39);
List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
Mat hierarchy = new Mat();
Imgproc.Canny(thresImage, binImage, 30, 10, 3, true); //контур
Imgproc.findContours(binImage, contours, hierarchy, Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE, new Point(0, 0));
hierarchy.release();
Imgproc.drawContours(binImage, contours, -1, new Scalar(255, 255, 255));//, 2, 8, hierarchy, 0, new Point());
MatOfPoint2f approxCurve = new MatOfPoint2f();
//For each contour found
for (int i = 0; i < contours.size(); i++) {
//Convert contours(i) from MatOfPoint to MatOfPoint2f
MatOfPoint2f contour2f = new MatOfPoint2f(contours.get(i).toArray());
//Processing on mMOP2f1 which is in type MatOfPoint2f
double approxDistance = Imgproc.arcLength(contour2f, true) * 0.02;
Imgproc.approxPolyDP(contour2f, approxCurve, approxDistance, true);
//Convert back to MatOfPoint
MatOfPoint points = new MatOfPoint(approxCurve.toArray());
// Get bounding rect of contour
Rect rect = Imgproc.boundingRect(points);
// draw enclosing rectangle (all same color, but you could use variable i to make them unique)
Imgproc.rectangle(binImage, new Point(rect.x, rect.y), new Point(rect.x + rect.width, rect.y + rect.height), new Scalar(255, 255, 255), 5);
}
And screen (not actually with processing values from code, just one with better results):
Original:
(unfortunately, I can't add more than 2 links to show more examples)
There were situations, when picture from this screen looked pretty good, but another pictures looked like with shapeless blobs.
Your code is fine, you just need to make a minor tweaks to get it work properly.
Firstly, the image size is very large, you can safely reduce it to 20% of current size without suffering a major loss in accuracy. Due to larger image size all the functions would perform slower.
You dont need to perform adaptive threshold before Canny, canny works perfectly on gray-scale images as well, You need to adjust the params as:
Canny(img, threshold1=170, threshold2=250)
which yields an image as:
[Optional] If you want to de-noise the image then you can try with morphological operations like erode and dilate.
Now you are ready to find the contours. The mistake in your code was using Imgproc.RETR_TREE flag you need to use Imgproc.RETR_EXTERNAL flag to get only the outer contours and not the nested inner contours.
At this step you may have some unwanted small contours, which can be filtered as:
// ** Below code if for reference purposes only, consult OpenCV docs for proper API methods
int character_area_lower_thresh = 10;
for (Contour c:contours) {
if (Imgproc.contourArea(c) > character_area_lower_thresh) {
// Desired contour, do what ever you want to do
Rect r = Imgproc.boundingRect(c);
}
}
My Question how to count Objects in image on Android Watershed OpenCv?
Since you didn't tell us which line causes those problem, but i strongly assume you misused erode() and dilate().
You should pass 3rd parameter of each function with valid non-zero Matrix.(e.g. Kernel)
erode & dilate example
// You can change the size of kennel
// if you want to change the range of erosion & dilation
Mat kelnel = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(2,2));
Mat fg = new Mat(img.size(),CvType.CV_8U);
Imgproc.erode(threeChannel, fg, kelnel);
Mat bg = new Mat(img.size(),CvType.CV_8U);
Imgproc.dilate(threeChannel, bg, kelnel);
I want to recognize numbers in the following image
I am currently using Tess4J library in eclipse java project but it only recognizes the characters in a plane color background. For this image it could not even identify that there are characters(numbers) on this image. Help me find a way to accomplish this task.
Here is my current code:
import net.sourceforge.tess4j.*;
import java.io.File;
public class Main {
public static void main(String[] args) {
File imageFile = new File("image.png");
Tesseract instance = Tesseract.getInstance();
try {
String result = instance.doOCR(imageFile);
System.out.println(result);
} catch (TesseractException e) {
System.err.println(e.getMessage());
}
}
}
and if there is way to count the squares separated by yellow lines.
Thank you
If your image is representative, then all you need as a first step is a binarization at a threshold close to the maximum value followed by discarding of small components.
f = Import["http://i.stack.imgur.com/6AXwH.jpg"]
step1 = SelectComponents[Binarize[ColorConvert[f, "Grayscale"], 0.9],
"Count", #1 > 100 &]
Now, if you know that the digits cannot be too tall or too thin (this is dependent on image dimensions), then you can filter the remaining components based on its bounding box.
SelectComponents[step1, "BoundingBox",
And[10 < #[[2, 1]] - #[[1, 1]] < 100, 50 < #[[2, 2]] - #[[1, 2]] < 100] &]
To separate each of the regions, you could consider using a colorspace where there is a channel dedicated to the yellow color. CMYK is a possibility here, and again all you need is a threshold at a high value, together with the basic morphological closing to complete the lines (since in your example the lines do not extend to the border of the image). Instead of using morphological closings here, you could detect the lines using Hough or RANSAC, for example.
rects = Closing[
Closing[Binarize[ColorSeparate[f, "CMYK"][[3]], 0.9],
ConstantArray[1, {1, 15}]], ConstantArray[1, {15, 1}]] (* left image *)
Colorize[MorphologicalComponents[ColorNegate[rects]],
ColorFunction -> "Rainbow"] (* right image *)
The tools used here are very simple, and almost any image processing library will provide them. There are also more robust approaches that could be taken, but for the given image it is not needed.
I'm trying to detect the positions of billiards balls on a table from an image taken at a perspective angle. I'm using the getPerspectiveTransform() method to find the transformation matrix and I want to apply that to only the circles I detect using HoughCircles. I'm trying to go from a rather large trapezoidal shape to a smaller rectangular shape. I don't want to do the transformation on the image first and then find the HoughCircles because the image gets too warped for houghcircles to provide useful results.
Here's my code:
CvMat mmat = cvCreateMat(3,3,CV_32FC1);
double srcX1 = 462;
double srcX2 = 978;
double srcX3 = 1440;
double srcX4 = 0;
double srcY = 241;
double srcHeight = 772;
double dstX = 56.8;
double dstY = 33.5;
double dstWidth = 262.4;
double dstHeight = 447.3;
CvSeq seq = cvHoughCircles(newGray, circles, CV_HOUGH_GRADIENT, 2.1d, (double)newGray.height()/40, 85d, 65d, 5, 50);
JavaCV.getPerspectiveTransform(new double[]{srcX1, srcY, srcX2,srcY, srcX3, srcHeight, srcX4, srcHeight},
new double[]{dstX, dstY, dstWidth, dstY, dstWidth, dstHeight, dstX, dstHeight}, mmat);
cvWarpPerspective(seq, seq, mmat);
for(int j=0; j<seq.total(); j++){
CvPoint3D32f point = new CvPoint3D32f(cvGetSeqElem(seq, j));
float xyr[] = {point.x(),point.y(),point.z()};
CvPoint center = new CvPoint(Math.round(xyr[0]), Math.round(xyr[1]));
int radius = Math.round(xyr[2]);
cvCircle(gray, center, 3, CvScalar.GREEN, -1, 8, 0);
cvCircle(gray, center, radius, CvScalar.BLUE, 3, 8, 0);
}
The problem is I get this error on the warpPerspective() method:
error: (-215) seq->total > 0 && CV_ELEM_SIZE(seq->flags) == seq->elem_size in function cv::Mat cv::cvarrToMat(const CvArr*, bool, bool, int)
Also I guess it's worth mentioning that I'm using JavaCV, in case the method calls look a bit different than what you're used to. Thanks for any help.
Answer:
the problem with what you want to do (besides the obvious, opencv wont let you) is that the radius cant really be warped correctly. AFAIK the xy coordinates are pretty easy to calculate x'=((m00x+m01y+m02)/(m20x+m21y+m22)) y'=((m10x+m11y+m12)/(m20x+m21y_m22)) when m is the transformation matrix. the radius you can hack by transforming all the points of the original circle and then find the max distance between x'y' and those points (atleast if the radius in the warped image is expected to cover all those points)
btw, mIJx = m(i,j)*x (just to clarify)
End Answer.
Everything i write is according to the c++ version, i've never used JavaCV but from what i could see its just a wrapper that calls the native c++ lib.
CvSeq is a sequance data structure that behaves like a linked list.
the assert your application crushes at is
CV_Assert(seq->total > 0 && CV_ELEM_SIZE(seq->flags) == seq->elem_size);
which means that either your seq instance is empty (total is the number of elements in the sequence) or somehow the inner seq flags are corrupted.
I'd recommend that you'd check the total member of your CvSeq, and the cvHoughCircles call.
all of this occurs before the actual implementation of cvWarpPerspective (its the first line in the implementation, that only converts your CvSeq to cv::Mat).. so its not the warping but what you're doing before that.
anyway, to understand whats wrong with cvHoughCircles we'll need more info about the creation of newGray and circles.
here is an example i've found on the javaCV page (Link)
IplImage gray = cvCreateImage( cvSize( img.width, img.height ), IPL_DEPTH_8U, 1 );
cvCvtColor( img, gray, CV_RGB2GRAY );
// smooth it, otherwise a lot of false circles may be detected
cvSmooth(gray,gray,CV_GAUSSIAN,9,9,2,2);
CvMemStorage circles = CvMemStorage.create();
CvSeq seq = cvHoughCircles(gray, circles.getPointer(), CV_HOUGH_GRADIENT,
2, img.height/4, 100, 100, 0, 0);
for(int i=0; i<seq.total; i++){
float xyr[] = cvGetSeqElem(seq,i).getFloatArray(0, 3);
CvPoint center = new CvPoint(Math.round(xyr[0]), Math.round(xyr[1]));
int radius = Math.round(xyr[2]);
cvCircle(img, center.byValue(), 3, CvScalar.GREEN, -1, 8, 0);
cvCircle(img, center.byValue(), radius, CvScalar.BLUE, 3, 8, 0);
}
from what i've seen in the implementation of cvHoughCircles, the answer is saved in the circles buff and at the end they create from it the CvSeq to return, so if you've allocated the circles buff wrong, it wont work.
EDIT:
as you can see, the CvSeq instance in case of the return from cvHoughCircles is a list of point-values, that is probably why the assertion failed. you cannot convert this CvSeq into a cv::Mat.. because its just not a cv::Mat. to get only the circles returned from cvHoughCircles in an cv::Mat instance, you'll need to create a new cv::Mat instance and than draw onto it all the circles in the CvSeq - as seen in the provided example above.
than the warping will work (you'll have a cv::Mat instance, and that is what the function expect - a cv::Mat as the only element in the CvSeq)
END EDIT
here is the c++ reference for CvSeq
and if you want to fiddle with the source code than
cvarrToMat is in matrix.cpp
CV_ELEM_SIZE is in types_c.h
cvWarpPerspective is in imgwarp.cpp
cvHoughCircles is in hough.cpp
I hope that will help.
BTW, your next error will probably be:
cv::warpPerspective in the C++ OpencCv asserts that dst.data != src.data
thus
cvWarpPerspective(seq, seq, mmat);
wont work cause your source mat and destination mat referencing the same data.
Not all the functions in OpenCV (and image processing in general) work in-situ (because there is no in-situ algorithm or because its slower then the other version eg. transpose of an n*n mat will work in-situ, but n*m where n!=m will be harder to do in-situ and might be slower)
you cant assume the using the src matrix as the dst will work.