We Keep Coding

Cropping an image with an outline

I have a captured image, the image consists of a table. I want to crop the table out of that image.
This is a sample image.
Can someone suggest what can be done?
I have to use it in android.

Use a hough transform to find the lines in the image.
OpenCV can easily do this and has java bindings. See the tutorial on this page on how to do something very similar.
https://docs.opencv.org/3.4.1/d9/db0/tutorial_hough_lines.html
Here is the java code provided in the tutorial:
import org.opencv.core.*;
import org.opencv.core.Point;
import org.opencv.highgui.HighGui;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;
class HoughLinesRun {
public void run(String[] args) {
// Declare the output variables
Mat dst = new Mat(), cdst = new Mat(), cdstP;
String default_file = "../../../../data/sudoku.png";
String filename = ((args.length > 0) ? args[0] : default_file);
// Load an image
Mat src = Imgcodecs.imread(filename, Imgcodecs.IMREAD_GRAYSCALE);
// Check if image is loaded fine
if( src.empty() ) {
System.out.println("Error opening image!");
System.out.println("Program Arguments: [image_name -- default "
+ default_file +"] \n");
System.exit(-1);
}
// Edge detection
Imgproc.Canny(src, dst, 50, 200, 3, false);
// Copy edges to the images that will display the results in BGR
Imgproc.cvtColor(dst, cdst, Imgproc.COLOR_GRAY2BGR);
cdstP = cdst.clone();
// Standard Hough Line Transform
Mat lines = new Mat(); // will hold the results of the detection
Imgproc.HoughLines(dst, lines, 1, Math.PI/180, 150); // runs the actual detection
// Draw the lines
for (int x = 0; x < lines.rows(); x++) {
double rho = lines.get(x, 0)[0],
theta = lines.get(x, 0)[1];
double a = Math.cos(theta), b = Math.sin(theta);
double x0 = a*rho, y0 = b*rho;
Point pt1 = new Point(Math.round(x0 + 1000*(-b)), Math.round(y0 + 1000*(a)));
Point pt2 = new Point(Math.round(x0 - 1000*(-b)), Math.round(y0 - 1000*(a)));
Imgproc.line(cdst, pt1, pt2, new Scalar(0, 0, 255), 3, Imgproc.LINE_AA, 0);
}
// Probabilistic Line Transform
Mat linesP = new Mat(); // will hold the results of the detection
Imgproc.HoughLinesP(dst, linesP, 1, Math.PI/180, 50, 50, 10); // runs the actual detection
// Draw the lines
for (int x = 0; x < linesP.rows(); x++) {
double[] l = linesP.get(x, 0);
Imgproc.line(cdstP, new Point(l[0], l[1]), new Point(l[2], l[3]), new Scalar(0, 0, 255), 3, Imgproc.LINE_AA, 0);
}
// Show results
HighGui.imshow("Source", src);
HighGui.imshow("Detected Lines (in red) - Standard Hough Line Transform", cdst);
HighGui.imshow("Detected Lines (in red) - Probabilistic Line Transform", cdstP);
// Wait and Exit
HighGui.waitKey();
System.exit(0);
}
}
public class HoughLines {
public static void main(String[] args) {
// Load the native library.
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
new HoughLinesRun().run(args);
}
}
Lines or LinesP will contain the found lines. Instead of drawing them (as in the example) you will want to manipulate them a little further.
Sort the found lines by slope.
The two largest clusters will be horizontal lines and then vertical lines.
For the horizontal lines calculate and sort by the y intercept.
The largest y intercept describes the top of the table.
The smallest y intercept is the bottom of the table.
For the vertical lines calculate and sort by the x intercept.
The largest x intercept is the right side of the table.
The smallest x intercept is the left side of the table.
You'll now have the coordinates of the four table corners and can do standard image manipulation to crop/rotate etc. OpenCV can help you will this step too.

Convert your image to grayscale.
Threshold your image to drop noise.
Find the minimum area rect of the non-blank pixels.
In python the code would look like:
import cv2
import numpy as np
img = cv2.imread('table.jpg')
imgray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
ret, thresh = cv2.threshold(imgray, 222, 255, cv2.THRESH_BINARY )
# write out the thresholded image to debug the 222 value
cv2.imwrite("thresh.png", thresh)
indices = np.where(thresh != 255)
coords = np.array([(b,a) for a, b in zip(*(indices[0], indices[1]))])
# coords = cv2.convexHull(coords)
rect = cv2.minAreaRect(coords)
box = cv2.boxPoints(rect)
box = np.int0(box)
cv2.drawContours(img, [box], 0, (0, 0, 255), 2)
cv2.imwrite("box.png", img)
For me this produces the following image.
If your image didn't have the red squares it would be a tighter fit.

findContours in opencv Java

if (DETECT_RED_OBJECTS_ONLY)
{
Imgproc.cvtColor(inputFrame, gray,
Imgproc.COLOR_BGR2RGB);
}
else
{
Imgproc.cvtColor(inputFrame, gray, Imgproc.COLOR_BGR2GRAY);
}
Imgproc.cvtColor(inputFrame, dst, Imgproc.COLOR_BGR2RGB);
// down-scale and upscale the image to filter out the noise
Imgproc.pyrDown(gray, downscaled, new Size(gray.cols() / 2, gray.rows() 2));
Imgproc.pyrUp(downscaled, upscaled, gray.size());
if (DETECT_RED_OBJECTS_ONLY) {
// convert the image from RGBA to HSV
Imgproc.cvtColor(upscaled, hsv, Imgproc.COLOR_RGB2HSV);
// threshold the image for the lower and upper HSV red range
Core.inRange(hsv, HSV_LOW_RED1, HSV_LOW_RED2, lowerRedRange);
Core.inRange(hsv, HSV_HIGH_RED1, HSV_HIGH_RED2, upperRedRange);
// put the two thresholded images together
Core.addWeighted(lowerRedRange, 1.0, upperRedRange, 1.0, 0.0, bw);
// apply canny to get edges only
System.out.println(bw);
Imgproc.Canny(bw, bw, 0, 255);
} else {
// Use Canny instead of threshold to catch squares with gradient shading
Imgproc.Canny(upscaled, bw, 0, 255);
}
// dilate canny output to remove potential
// holes between edge segments
Imgproc.dilate(bw, bw, new Mat(), new Point(-1, 1), 1);
// find contours and store them all as a list
List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
contourImage = bw.clone();
Imgproc.findContours(
contourImage,
contours,
hierarchyOutputVector,
Imgproc.RETR_EXTERNAL,
Imgproc.CHAIN_APPROX_SIMPLE
);
System.out.println("contours" + contours);
// loop over all found contours
for (MatOfPoint cnt : contours) {
MatOfPoint2f curve = new MatOfPoint2f(cnt.toArray());
// approximates a polygonal curve with the specified precision
Imgproc.approxPolyDP(
curve,
approxCurve,
0.02 * Imgproc.arcLength(curve, true),
true
);
int numberVertices = (int) approxCurve.total();
When I use triangle as an input image , this one works and detect triangle[counters become 1 and numberVertices becomes 3]. But when i input a rectangle image , contours become 2 and numberVertices becomes 2. It must be 4 . Isn't it. Can anyone help me to find the bug.

If you are using this in latest visual studio, I suggest you to use :
JavaList List<MatOfPoint> contours = new JavaList<MatOfPoint>();
this might change the behavior of your contours. Please let me know if this fixed your bug. Otherwise lets try with something else

Using OpenCV to find the bounding box of numbers on an image

I'm trying to find the bounding box of the numbers in the middle of the below 3 images.
Here's 3 example cards I'm trying to work with.
The code I'm using is based (almost a complete copy) of the code provided in the first answer here, although converted to Java (answers in C++ are fine) and added parameters for the size of the contours to merge (defined as sizeHorizonal and sizeVertical in my code), which are the two parameters I'm playing with in the images below.
MatOfPoint2f approxCurve = new MatOfPoint2f();
Mat imgMAT = new Mat();
Utils.bitmapToMat(bmp32, imgMAT);
Mat grad = new Mat();
Imgproc.cvtColor(imgMAT, grad, Imgproc.COLOR_BGR2GRAY);
Mat img_sobel = new Mat();
Mat img_threshold = new Mat();
Imgproc.Sobel(grad, img_sobel, CvType.CV_8U, 1, 0, 3, 1, 0, Core.BORDER_DEFAULT);
Imgproc.threshold(img_sobel, img_threshold, 0, 255, Imgproc.THRESH_OTSU + Imgproc.THRESH_BINARY);
Mat element = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(sizeHorizontal, sizeVertical));
Imgproc.morphologyEx(img_threshold, img_threshold, Imgproc.MORPH_CLOSE, element);
Imgproc.cvtColor(imgMAT, imgMAT, Imgproc.COLOR_BGR2GRAY);
List<MatOfPoint> contours = new ArrayList<>();
Imgproc.findContours(img_threshold, contours, new Mat(), Imgproc.RETR_CCOMP, Imgproc.CHAIN_APPROX_SIMPLE, new org.opencv.core.Point(0, 0));
for (int i = 0; i < contours.size(); i++) {
//Convert contours(i) from MatOfPoint to MatOfPoint2f
MatOfPoint2f contour2f = new MatOfPoint2f( contours.get(i).toArray() );
//Processing on mMOP2f1 which is in type MatOfPoint2f
double approxDistance = Imgproc.arcLength(contour2f, true)*0.02;
Imgproc.approxPolyDP(contour2f, approxCurve, approxDistance, true);
//Convert back to MatOfPoint
MatOfPoint points = new MatOfPoint( approxCurve.toArray() );
// Get bounding rect of contour
org.opencv.core.Rect rect = Imgproc.boundingRect(points);
Imgproc.rectangle(imgMAT, rect.tl(), rect.br(), new Scalar(0, 255, 0), 2);
}
I've got the separate number sections contoured, but I can't find a way to isolate the contours I want. Here's the areas contoured with the parameters for the size input. As you can see for the second image, this is working exactly as I want and has contoured the whole number, rather than each section.
1: Size param input: 17, 5
2: Size param input: 23, 7
3: Size param input: 23, 13
So, things I need help with:
Isolating the four contours in the middle and finding a way to merge these contours together
I've thought about taking the contours that match a given aspect ratio and cropping to a bounding box encompassing all of them, but there are other surrounding contours with similar ratios.
Finding a way to choose the correct size parameters automatically (as each card type requires different parameters to isolate the numbers)
Short of trying all 3 size inputs and seeing what gives the expected contours, I could use the prevailing colour as an indicator of the card type and then use the parameters for this card type. But, any other suggestions would be helpful, as I feel there's a better way to do this.
Many thanks!

Out of my busy schedule i could help you to some extinct. Please find the code below which will help you for first two images. Fine tune it for the third image. Just play with morphological operations to get the required output.
//#include "stdafx.h"
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <iostream>
#include "tchar.h"
using namespace cv;
using namespace std;
#define INPUT_FILE "p.jpg"
#define OUTPUT_FOLDER_PATH string("")
int _tmain(int argc, _TCHAR* argv[])
{
Mat large = imread(INPUT_FILE);
Mat rgb;
// downsample and use it for processing
pyrDown(large, rgb);
Mat small;
cvtColor(rgb, small, CV_BGR2GRAY);
// morphological gradient
Mat grad;
Mat morphKernel = getStructuringElement(MORPH_ELLIPSE, Size(2, 2));
Mat morphKernel1 = getStructuringElement(MORPH_ELLIPSE, Size(1, 1));
morphologyEx(small, grad, MORPH_GRADIENT, morphKernel);
// binarize
Mat bw;
threshold(grad, bw, 5.0, 50.0, THRESH_BINARY | THRESH_OTSU);
// connect horizontally oriented regions
Mat connected;
morphKernel = getStructuringElement(MORPH_RECT, Size(9, 1));
morphologyEx(bw, connected, MORPH_CLOSE, morphKernel);
morphologyEx(bw, connected, MORPH_OPEN, morphKernel1);
morphologyEx(connected, connected, MORPH_CLOSE, morphKernel);
morphologyEx(connected, connected, MORPH_CLOSE, morphKernel);
morphologyEx(connected, connected, MORPH_CLOSE, morphKernel);
// find contours
Mat mask = Mat::zeros(bw.size(), CV_8UC1);
vector<vector<Point>> contours;
vector<Vec4i> hierarchy;
findContours(connected, contours, hierarchy, CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE, Point(0, 0));
// filter contours
int y=0;
for(int idx = 0; idx >= 0; idx = hierarchy[idx][0])
{
Rect rect = boundingRect(contours[idx]);
Mat maskROI(mask, rect);
maskROI = Scalar(0, 0, 0);
// fill the contour
drawContours(mask, contours, idx, Scalar(255, 255, 255), CV_FILLED);
double a=contourArea( contours[idx],false);
if(a> 575)
{
rectangle(rgb, rect, Scalar(0, 255, 0), 2);
y++;
}
imshow("Result1",rgb);
}
cout<<" The number of elements"<<y<< endl;
imshow("Result",mask);
imwrite(OUTPUT_FOLDER_PATH + string("rgb.jpg"), rgb);
waitKey(0);
return 0;
}

Trouble detecting skin

I'm trying to detect skin on images using opencv (new at it) but I've managed to get SOME of the skin to be detected, however the rest seems to cause some noise on the image. Here's the original image:
The result of my code is:
The code that prouduces this:
Mat image = Imgcodecs.imread(name);
Imgproc.pyrMeanShiftFiltering(image, image, 10, 20);
Imgproc.blur(image, image, new Size(3, 3));
Mat hsv = new Mat();
Imgproc.cvtColor(image, hsv, Imgproc.COLOR_BGR2HSV);
Mat bw = new Mat();
Core.inRange(hsv, new Scalar(0, 10, 60), new Scalar(20, 150, 255), bw);
List<MatOfPoint> contours = new ArrayList<>();
Mat hierarchy = new Mat();
Imgproc.findContours(bw, contours, hierarchy, Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_NONE, new Point(0, 0));
int s = findBiggestContour(contours);
Mat drawing = new Mat(image.size(), CvType.CV_8UC1);
Imgproc.drawContours(drawing, contours, s, new Scalar(255), -1, 8, hierarchy, 0, new Point(0, 0));
Imgcodecs.imwrite(rename(name), drawing);
How do I fix the code to detect the remaining skin on the image and get rid of the noise?
I'm using Java with OpenCV 3.0.0.

Since you're using the findBiggestConour() I think you only draw the biggest match, not all of them. Since the biggest contour happens to be the one on the second image only that is shown.

Just to add to what JanSLO said here above, I tried your code and instead of drawing just the biggest contour, I drew all contours and got the following result.
//c++ code, not java
Mat drawing = Mat::zeros(img.size(), CV_8UC1 );
for(int i=0; i < contours.size(); ++i) {
drawContours(drawing, contours, i, Scalar(255), 3, 8, hierarchy, 0, Point(0, 0));
}
imwrite("data/skin_detect_out.jpg", drawing);
I am pleasantly surprised at the result, since this is such a simple piece of code. More advanced pixel based skin detection methods involve making a probability model of skin pixels using training data, and using that model to classify whether a given pixel is skin or not.

Using warpPerspective() on a sequence of points given by HoughCircles(), OpenCV

I'm trying to detect the positions of billiards balls on a table from an image taken at a perspective angle. I'm using the getPerspectiveTransform() method to find the transformation matrix and I want to apply that to only the circles I detect using HoughCircles. I'm trying to go from a rather large trapezoidal shape to a smaller rectangular shape. I don't want to do the transformation on the image first and then find the HoughCircles because the image gets too warped for houghcircles to provide useful results.
Here's my code:
CvMat mmat = cvCreateMat(3,3,CV_32FC1);
double srcX1 = 462;
double srcX2 = 978;
double srcX3 = 1440;
double srcX4 = 0;
double srcY = 241;
double srcHeight = 772;
double dstX = 56.8;
double dstY = 33.5;
double dstWidth = 262.4;
double dstHeight = 447.3;
CvSeq seq = cvHoughCircles(newGray, circles, CV_HOUGH_GRADIENT, 2.1d, (double)newGray.height()/40, 85d, 65d, 5, 50);
JavaCV.getPerspectiveTransform(new double[]{srcX1, srcY, srcX2,srcY, srcX3, srcHeight, srcX4, srcHeight},
new double[]{dstX, dstY, dstWidth, dstY, dstWidth, dstHeight, dstX, dstHeight}, mmat);
cvWarpPerspective(seq, seq, mmat);
for(int j=0; j<seq.total(); j++){
CvPoint3D32f point = new CvPoint3D32f(cvGetSeqElem(seq, j));
float xyr[] = {point.x(),point.y(),point.z()};
CvPoint center = new CvPoint(Math.round(xyr[0]), Math.round(xyr[1]));
int radius = Math.round(xyr[2]);
cvCircle(gray, center, 3, CvScalar.GREEN, -1, 8, 0);
cvCircle(gray, center, radius, CvScalar.BLUE, 3, 8, 0);
}
The problem is I get this error on the warpPerspective() method:
error: (-215) seq->total > 0 && CV_ELEM_SIZE(seq->flags) == seq->elem_size in function cv::Mat cv::cvarrToMat(const CvArr*, bool, bool, int)
Also I guess it's worth mentioning that I'm using JavaCV, in case the method calls look a bit different than what you're used to. Thanks for any help.

Answer:
the problem with what you want to do (besides the obvious, opencv wont let you) is that the radius cant really be warped correctly. AFAIK the xy coordinates are pretty easy to calculate x'=((m00x+m01y+m02)/(m20x+m21y+m22)) y'=((m10x+m11y+m12)/(m20x+m21y_m22)) when m is the transformation matrix. the radius you can hack by transforming all the points of the original circle and then find the max distance between x'y' and those points (atleast if the radius in the warped image is expected to cover all those points)
btw, mIJx = m(i,j)*x (just to clarify)
End Answer.
Everything i write is according to the c++ version, i've never used JavaCV but from what i could see its just a wrapper that calls the native c++ lib.
CvSeq is a sequance data structure that behaves like a linked list.
the assert your application crushes at is
CV_Assert(seq->total > 0 && CV_ELEM_SIZE(seq->flags) == seq->elem_size);
which means that either your seq instance is empty (total is the number of elements in the sequence) or somehow the inner seq flags are corrupted.
I'd recommend that you'd check the total member of your CvSeq, and the cvHoughCircles call.
all of this occurs before the actual implementation of cvWarpPerspective (its the first line in the implementation, that only converts your CvSeq to cv::Mat).. so its not the warping but what you're doing before that.
anyway, to understand whats wrong with cvHoughCircles we'll need more info about the creation of newGray and circles.
here is an example i've found on the javaCV page (Link)
IplImage gray = cvCreateImage( cvSize( img.width, img.height ), IPL_DEPTH_8U, 1 );
cvCvtColor( img, gray, CV_RGB2GRAY );
// smooth it, otherwise a lot of false circles may be detected
cvSmooth(gray,gray,CV_GAUSSIAN,9,9,2,2);
CvMemStorage circles = CvMemStorage.create();
CvSeq seq = cvHoughCircles(gray, circles.getPointer(), CV_HOUGH_GRADIENT,
2, img.height/4, 100, 100, 0, 0);
for(int i=0; i<seq.total; i++){
float xyr[] = cvGetSeqElem(seq,i).getFloatArray(0, 3);
CvPoint center = new CvPoint(Math.round(xyr[0]), Math.round(xyr[1]));
int radius = Math.round(xyr[2]);
cvCircle(img, center.byValue(), 3, CvScalar.GREEN, -1, 8, 0);
cvCircle(img, center.byValue(), radius, CvScalar.BLUE, 3, 8, 0);
}
from what i've seen in the implementation of cvHoughCircles, the answer is saved in the circles buff and at the end they create from it the CvSeq to return, so if you've allocated the circles buff wrong, it wont work.
EDIT:
as you can see, the CvSeq instance in case of the return from cvHoughCircles is a list of point-values, that is probably why the assertion failed. you cannot convert this CvSeq into a cv::Mat.. because its just not a cv::Mat. to get only the circles returned from cvHoughCircles in an cv::Mat instance, you'll need to create a new cv::Mat instance and than draw onto it all the circles in the CvSeq - as seen in the provided example above.
than the warping will work (you'll have a cv::Mat instance, and that is what the function expect - a cv::Mat as the only element in the CvSeq)
END EDIT
here is the c++ reference for CvSeq
and if you want to fiddle with the source code than
cvarrToMat is in matrix.cpp
CV_ELEM_SIZE is in types_c.h
cvWarpPerspective is in imgwarp.cpp
cvHoughCircles is in hough.cpp
I hope that will help.
BTW, your next error will probably be:
cv::warpPerspective in the C++ OpencCv asserts that dst.data != src.data
thus
cvWarpPerspective(seq, seq, mmat);
wont work cause your source mat and destination mat referencing the same data.
Not all the functions in OpenCV (and image processing in general) work in-situ (because there is no in-situ algorithm or because its slower then the other version eg. transpose of an n*n mat will work in-situ, but n*m where n!=m will be harder to do in-situ and might be slower)
you cant assume the using the src matrix as the dst will work.