I'm trying to convert a 3d point in OpenGL to a 2d point on screen to render a healthbar for a little game I'm writing. However, I'm having some trouble retrieving the x coordinate of where to draw the healthbar. Basically, the healthbar must appear to be above a player, but must always have the same width/height relative to the screen.
I tweaked a snippet of code I found from the accepted answer at Convert a 3D location to a 2D on-screen point. (XYZ => XY) and I now have this
public static int[] getScreenCoords(double x, double y, double z) {
FloatBuffer screenCoords = BufferUtils.createFloatBuffer(4);
IntBuffer viewport = BufferUtils.createIntBuffer(16);
FloatBuffer modelView = BufferUtils.createFloatBuffer(16);
FloatBuffer projection = BufferUtils.createFloatBuffer(16);
// int[] screenCoords = new double[4];
// int[] viewport = new int[4];
// double[] modelView = new double[16];
// double[] projection = new double[16];
GL11.glGetFloat(GL11.GL_MODELVIEW_MATRIX, modelView);
GL11.glGetFloat(GL11.GL_PROJECTION_MATRIX, projection);
GL11.glGetInteger(GL11.GL_VIEWPORT, viewport);
boolean result = GLU.gluProject((float) x, (float) y, (float) z, modelView, projection, viewport, screenCoords);
if (result) {
return new int[] { (int) screenCoords.get(3), (int) screenCoords.get(1) };
}
return null;
}
It seems to work fine with the y coordinate, however, x always returns 0 no matter what the angle is.
Many thanks in advance!
screenCoords.get(3) should be screenCoords.get(0) because the x position is stored at index 0. You also only actually need the capacity of screenCoords to be 3 floats, not 4.
Related
I'm trying to convert a 3d vector to a 2d vector in LWJGL 3.
The goal is to render a nametag on the 2d screen while moving in a 3d world.
this is what I've used on LWJGL 2:
public static Vector2d to2D(double x, double y, double z) {
FloatBuffer screenCoords = BufferUtils.createFloatBuffer(3);
IntBuffer viewport = BufferUtils.createIntBuffer(16);
FloatBuffer modelView = BufferUtils.createFloatBuffer(16);
FloatBuffer projection = BufferUtils.createFloatBuffer(16);
GL11.glGetFloat(GL11.GL_MODELVIEW_MATRIX, modelView);
GL11.glGetFloat(GL11.GL_PROJECTION_MATRIX, projection);
GL11.glGetInteger(GL11.GL_VIEWPORT, viewport);
boolean result = GLU.gluProject((float) x, (float) y, (float) z, modelView, projection, viewport, screenCoords);
if (result) {
return new Vector2d(screenCoords.get(0), Display.getHeight() - screenCoords.get(1));
}
return null;
}
In LWJGL 2 it was doable using GLU.gluProject() which is now removed, is there any alternative in the new LWJGL version?
You can use JOML (which is also downloadable as an LWJGL 3 Addon) and its Matrix4f.project() method for this. The following is a functioning port of your shown code snippet for LWJGL3 + JOML:
import java.nio.*;
import org.joml.*;
import org.lwjgl.opengl.GL11;
import org.lwjgl.system.MemoryStack;
public class Snippet {
public static Vector2d to2D(double x, double y, double z, int displayHeight) {
Vector3f screenCoords = new Vector3f();
int[] viewport = new int[4];
try (MemoryStack stack = MemoryStack.stackPush()) {
FloatBuffer modelView = stack.mallocFloat(16);
FloatBuffer projection = stack.mallocFloat(16);
GL11.glGetFloatv(GL11.GL_MODELVIEW_MATRIX, modelView);
GL11.glGetFloatv(GL11.GL_PROJECTION_MATRIX, projection);
GL11.glGetIntegerv(GL11.GL_VIEWPORT, viewport);
new Matrix4f(projection)
.mul(new Matrix4f(modelView))
.project((float) x, (float) y, (float) z, viewport, screenCoords);
}
return new Vector2d(screenCoords.x, displayHeight - screenCoords.y);
}
}
On top of the port, I've also followed best practices regarding native memory management in LWJGL 3: https://blog.lwjgl.org/memory-management-in-lwjgl-3/#strategy5
I'm currently trying to register touches on the screen in World Space.
I first convert them to normalized Device Coordinates and then try to multiply a point at the near side of the normalized cube (z = -1) and a point at the far side of the normalized cube (z = 1) with the inverted ProjectionViewMatrix to get a Line between them.
My approach so far:
//Calculate ProjectionViewMatrix
Matrix.multiplyMM(projectionViewMatrix,0,perspectiveProjectionMatrix,0,viewMatrix,0);
//Calculate Inverse
Matrix.invertM(invertedProjectionViewMatrix,0,projectionViewMatrix,0);
float[] nearPoint = {x, y, -1, 1};
float[] farPoint = {x, y, 1, 1};
float[] nearPointWorldSpace = new float[4];
float[] farPointWorldSpace = new float[4];
Matrix.multiplyMV(nearPointWorldSpace,0, invertedProjectionViewMatrix,0, nearPoint,0);
Matrix.multiplyMV(farPointWorldSpace,0, invertedProjectionViewMatrix,0, farPoint,0);
perspectiveDevide(nearPointWorldSpace);
perspectiveDevide(farPointWorldSpace);
Where perspectiveDevide is defined as:
private static void perspectiveDevide(float[] vector) {
vector[0] /= vector[3];
vector[1] /= vector[3];
vector[2] /= vector[3];
}
Now what I should get is a near and far point that have the same or very similar X/Y-Coordinates, because my Camera is right above the lookAt and with no angle.
However what I do get is this:
NearPointWorld:
[0] -0.002805814
[1] 0.046295937
[2] 1.9
[3] 9.999999
FarPointWorld:
[0] -2.8057494
[1] 46.294872
[2] -97.99771
[3] 0.010000229
Any Ideas what might be wrong?
EDIT:
Here's my code for the View and Projection Matrix:
Projection:
Matrix.perspectiveM(perspectiveProjectionMatrix,0, 60, (float) width / (float) height, 0.1f, 100f);
View:
Matrix.setLookAtM(viewMatrix,0,
0,0,2,
0,0,0,
0,1,0);
As Nico Schertler pointed out, these results are actuallly reasonable.
To get the correct X/Y Coordinates I had to unproject the screen center.
I have problem using Vuforia with jPCT.
I have successfully passed the modelViewMatrix from Vuforia native code
QCAR::Matrix44F modelViewMatrix = QCAR::Tool::convertPose2GLMatrix(imageResult->getPose())
to Java.
And then I try to set the camera matrix of jPCT.
public void setCameraMatrix(float[] modelViewMatrixFromVuforia) {
float x = modelViewMatrixFromVuforia[12];
float y = modelViewMatrixFromVuforia[13];
float z = modelViewMatrixFromVuforia[14];
modelViewMatrixFromVuforia[12] = 0;
modelViewMatrixFromVuforia[13] = 0;
modelViewMatrixFromVuforia[14] = 0;
Matrix cameraMatrix = new Matrix();
cameraMatrix.setDump(modelViewMatrixFromVuforia);
cameraMatrix = cameraMatrix.invert();
camera.setBack(cameraMatrix);
camera.setPosition(x, y, z);
}
But the 3D Object is not being tracked properly. What have I missed?
I'm using this and it works perfectly:
private Matrix mMatrix = new Matrix();
...
mMatrix.setDump(modelViewMatrixFromVuforia); // float[16] sent from native code
mCamera.setBack(mMatrix);
But you have to rotate the matrix 180 degree around X axis before you send it to Java in order to match the coordinate system from Vuforia to jPCT.
Do the rotation in native codes as follows:
SampleUtils::rotatePoseMatrix(180.0f, 1.0f, 0, 0, &modelViewMatrix.data[0]);
I'm trying to use gluProject with my game. I've found a way to use it but I don't know how to deal with rotation.
Here is the code I made:
public static Vector2 getScreenCoordinates(float x, float y, float z, int height, int width)
{
FloatBuffer screen_coords = GLAllocation.createDirectFloatBuffer(4);
IntBuffer viewport = GLAllocation.createDirectIntBuffer(16);
FloatBuffer modelview = GLAllocation.createDirectFloatBuffer(16);
FloatBuffer projection = GLAllocation.createDirectFloatBuffer(16);
GL11.glGetFloat(2982, modelview);
GL11.glGetFloat(2983, projection);
GL11.glGetInteger(2978, viewport);
boolean result = GLU.gluProject(x, y, z, modelview, projection, viewport, screen_coords);
if (result)
{
float screen_x = screen_coords.get(0);
float screen_y = screen_coords.get(1);
float scrren_z = screen_coords.get(2);
screen_y -= height / 2;
screen_y = -screen_y;
screen_y += height / 2;
return new Vector2(screen_x, screen_y);
}
else
{
System.out.printf("Failed to convert 3D coords to 2D screen coords");
return null;
}
}
So x, y and z are coords in my 3D map. height and width are my window size.
How can I modify it to deal with rotation (yaw and pitch)?
Thanks.
Here is my new code:
public Vector2 worldToScreen(double x, double y, double z, int yaw, int pitch)
{
// Initialize the result buffer
FloatBuffer screen_coords = GLAllocation.createDirectFloatBuffer(4);
// Init the OpenGL buffers
IntBuffer viewport = GLAllocation.createDirectIntBuffer(16);
FloatBuffer modelview = GLAllocation.createDirectFloatBuffer(16);
FloatBuffer projection = GLAllocation.createDirectFloatBuffer(16);
// Add the rotation
GL11.glRotatef(yaw, 0, 0, 1);
GL11.glRotatef(pitch, 1, 0, 0);
// Get the OpenGL data
GL11.glGetFloat(2982, modelview);
GL11.glGetFloat(2983, projection);
GL11.glGetInteger(2978, viewport);
// Remove the rotation
GL11.glRotatef(-yaw, 0, 0, 1);
GL11.glRotatef(-pitch, 1, 0, 0);
// Calculate the screen position
boolean result = GLU.gluProject(x, y, z, modelview, projection, viewport, screen_coords);
if (result)
{
if ( (screen_coords.get(0) < -1.0f) || (screen_coords.get(0) > 1.0f) || (screen_coords.get(1) < -1.0f) || (screen_coords.get(1) > 1.0f) )
return null;
int window_half_width = getWidth() / 2;
int window_half_height = getHeight() / 2;
int screen_x = window_half_width + (window_half_width * -screen_coords.get(0));
int screen_y = window_half_height + (window_half_height * screen_coords.get(1));
System.out.printf("(Full Screen / No bounds) [" +x+ ", " +y+ ", " +z+ "] gives [" +screen_coords.get(0)+ ", " +screen_coords.get(1)+ "]");
System.out.printf("(Every Time) [" +x+ ", " +y+ ", " +z+ "] gives [" +screen_x+ ", " +screen_y+ "]");
return new Vector2(screen_x, screen_y);
}
else
{
System.out.printf("Failed to convert 3D coords to 2D screen coords");
return null;
}
}
Is that correct?
Yaw is a rotation about the y axis (pointing upwards), and pitch is a rotation about the x axis.
GL11.glRotatef(pitch, 1, 0, 0);
GL11.glRotatef(yaw, 0, 1, 0);
It may also be faster to push/pop the the projection matrix, and you should be in the correct matrix mode. (Both your model-view matrix and your projection matrix have to be correct).
GL11.glPushMatrix();
// Camera rotations
GL11.glPopMatrix();
But it's questionable as to why you need to undo the camera tranformations. You should probably just apply them once per frame or viewport. Render your scene with that view, then get the projected coordinate and then change to your 2D rendering mode.
Update
I'd prefer it if you used roll, pitch and yaw of the camera to mean what they actually mean in OpenGL's coordinate system - with respect to the camera. The point is, as long as you have the camera model-view and projection matrices setup correctly, you can project the point. You must already be setting up the camera somewhere. You will be able to see if this is correct, because you can SEE the results. Apply that transformation and you will know it's correct.
Is this correct? It might be? You have to apply all of the transformations. The exact same transformations that give you the camera view you see in game. This includes all translations and rotations. You will know if you are right, because you should already be applying these transformations in-game.
However, What you need to think about is what is the camera projection matrix before hand? If you don't know, call glLoadIdentity() to clear it, and then call your camera transformation logic. You need an accurate model-view and projection matrix - the exact same camera setup you are using in-game. If your model-view or projection matrices are in the wrong state, it just won't work.
I can't get the circle vectors values from the OpenCV Mat in Android. I want to use this function:
HoughCircles(Mat image, Mat circles, int method, double dp, double minDist)
And then I want to show the circles that were found. Where I'm stuck is how to use the circles parameter in this function.
So, the question is: how can I get numbers of 3-element vectors and values of every element in vector from Mat type of OpenCV in Android?
Once you have your circles Mat
for (int i = 0; i < circles.cols(); i++)
{
double vCircle[] = circles.get(0,i);
double x = vCircle[0];
double y = vCircle[1];
double radius = vCircle[2];
}
Ideally you would want to use a vector<Vec3f> list to process the circles like this:
vector<Vec3f> circles;
// do HoughCircles...
for(size_t i = 0; i < circles.size(); i++)
{
Vec3f circle = circles[i];
Point2f center(circle[0] /* x */, circle[1] /* y */);
float radius = circle[2];
// use the circle...
}
EDIT : I tried the code just using a Mat, and it appears that the circle parameters are stored as a 1xN matrix with elements of type CV_32FC3, and where N is the number of circles detected. So, each column contains the (x, y, radius) vector you need.
Here is a sample I wrote in C++ showing this:
Mat circleImage = imread("circle.png", 0);
Mat circleDisp;
cvtColor(circleImage, circleDisp, CV_GRAY2RGB);
Mat circles;
HoughCircles(circleImage, circles, CV_HOUGH_GRADIENT, 2, circleImage.rows >> 2, 200, 100);
for( size_t i = 0; i < circles.cols; i++ )
{
Vec3f vCircle = circles.at<Vec3f>(i);
Point center(cvRound(vCircle[0]), cvRound(vCircle[1]));
int radius = cvRound(vCircle[2]);
// draw the circle center
circle( circleDisp, center, 3, Scalar(0,255,0), -1, 8, 0 );
// draw the circle outline
circle( circleDisp, center, radius, Scalar(0,0,255), 3, 8, 0 );
}
namedWindow( "circles", 1 );
imshow( "circles", circleDisp );
waitKey();
Hope that helps!
Just cast your Mat to vector:
HoughCircles(Mat image, Mat circles, int method, double dp, double minDist);
vector<Vec3f> myCircles = (vector<Vec3f>)circles;
Or, simpler
HoughCircles(Mat image, vector<Vec3f>& circles,
int method, double dp, double minDist);
Note
This is only true for OpenCV 2.3.1.