We Keep Coding

glTranslatef and mouse clicks - gluUnProject

Please see bottom of question for the current solution I have gone for, thanks to Finlaybob, elect, gouessej
An appeal to the Elders of OpenGL.... I am having big problems with detecting the relative position of a mouse click on my textured plane.
I am making a game where I am drawing a single large square and texturing it with a large generated map texture. The view is always top down and you can only currently move the X Y and Z coordinates of that square.
Screenshot of the map
OpenGL init
screenRatio = (float)screenW / (float)screenH;
System.out.println("init");
glu = new GLU();
GL2 gl2 = drawable.getGL().getGL2();
gl2.glShadeModel( GL2.GL_SMOOTH );
gl2.glHint( GL2.GL_PERSPECTIVE_CORRECTION_HINT, GL2.GL_NICEST );
gl2.glClearColor( 0f, 0f, 0f, 1f );
gl2.glDepthMask(false);
gl2.glEnable(GL2.GL_DEPTH_TEST);
Set camera position
gl2.glViewport(0, 0, 1024, 768);
gl2.glMatrixMode( GL2.GL_PROJECTION );
gl2.glLoadIdentity();
glu.gluPerspective( 45, screenRatio, 1, 100 );
glu.gluLookAt( 0, 0, 3, 0, 0, 0, 0, 1, 0 );
gl2.glMatrixMode(GL2.GL_MODELVIEW);
gl2.glLoadIdentity();
Move position to start drawing the map
// typical camera coord example:
// CENTRE: 0.0f, 0.0f, 10f
// FULL ZOOM OUT AND TOP LEFT: -25f, 25f, 40f
// move position
gl2.glTranslatef( -cameraX, -cameraY, -cameraZ );
I suspect the glTranslatef z-coord may be a suspect. As I am drawing the square 40f ( for example ) away from the origin
Map vertex information
// here are the coordinates/dimensions of my textured square ( my map )
float[] vertexArray = {
-25f, 25f,
25f, 25f,
25f, -25f,
25f, -25f,
};
Mouse click position calculation
"Borrowed" from java-tips 1628-how-to-use-gluunproject-in-jogl.html
int x = mouse.getX(), y = mouse.getY();
int viewport[] = new int[4];
double mvmatrix[] = new double[16];
double projmatrix[] = new double[16];
int realy = 0;
double wcoord[] = new double[4];
gl2.glGetIntegerv(GL2.GL_VIEWPORT, viewport, 0);
gl2.glGetDoublev(GL2.GL_MODELVIEW_MATRIX, mvmatrix, 0);
gl2.glGetDoublev(GL2.GL_PROJECTION_MATRIX, projmatrix, 0);
realy = viewport[3] - (int) y - 1;
glu.gluUnProject(
(double) x,
(double) realy,
0.0, // I have experimented with having this as 1.0 also
mvmatrix, 0,
projmatrix, 0,
viewport, 0,
wcoord, 0
);
Experimenting with the near/far bit ( 3rd param of gluUnProject ) seems to produce a better effect but there seems to be no sweet spot ( the best I found was 0.945 )
I would very much like mCX, mCY to be relative to the rendered map coordinates ( -25f - 25f ) regardless of Z position
mCX = (float)wcoord[0];
mCY = (float)wcoord[1];
Draw a rectangle at the translated coordinates
gl2.glColor3f(1.f, 0.f, 0.f);
gl2.glBegin(GL2.GL_QUADS);
gl2.glVertex2f( mCX-0.1f, mCY+0.1f );
gl2.glVertex2f( mCX+0.1f, mCY+0.1f );
gl2.glVertex2f( mCX+0.1f, mCY-0.1f );
gl2.glVertex2f( mCX-0.1f, mCY-0.1f );
gl2.glEnd();
Currently the coordinates work well in relation to x & y translation, if I click the very centre of the screen it will draw a box approximately in the correct place regardless of my glTranslatef movement. If I click away from the centre of the screen I see an exponential offset.
Demonstration of exponential offset
When I click the very dead centre of the screen it will draw this mauve square exactly around the mouse point, but with the smallest of movement it will create the following effect:
Fully zoomed in, click a couple of pixels right of centre
UPDATE AND WORKING... FOR NOW
At the time of generating the texture for my map I also generate an alternative texture which represents each "tile" as a different colour. In my initial and current attempt the colour of this tile is a function of it's X and Y coordinates ( a map is made up of 100 tiles across and 100 tiles down, so the x+y coordinates range from 0 - 99 )
I end up with a texture which looks like a gradient from green to red. The below code will, at the time of a mouse click, quickly render this texture ( imperceptible to user ) and read the rgb value under the mouse. We then turn that rgb value into a world coordinate and BOOM... the relative coordinates of my map are realised.
float pX, pY;
// render a colourised version of the scene for the purposes of "picking"
// https://www.opengl.org/archives/resources/faq/technical/selection.htm
public void pick ( GL2 gl2 ) {
// DRAW PICKING SCENE
gl2.glClearBufferfv(GL2.GL_COLOR, 0, clearColor);
gl2.glClearBufferfv(GL2.GL_DEPTH, 0, clearDepth);
gl2.glTranslatef( -cameraX, -cameraY, -cameraZ );
// draw my map but use the colour gradient texture
for ( Entity e : this.entities ) {
e.drawPick( gl2 );
}
// not sure what this does #cargo-cult
gl2.glFlush();
gl2.glFinish();
gl2.glPixelStorei(GL2.GL_UNPACK_ALIGNMENT, 1);
// After rendering ask OpenGL to read the colour of the screen at the given window coordinates!
FloatBuffer buffer = FloatBuffer.allocate(4);
int realy = 0;
int viewport[] = new int[4];
gl2.glGetIntegerv(GL2.GL_VIEWPORT, viewport, 0);
realy = viewport[3] - (int) mouse.getY() - 1;
gl2.glReadPixels( mouse.getX(), realy, 1, 1, GL2.GL_RGBA, GL2.GL_FLOAT, buffer);
float[] pixels = new float[3];
pixels = buffer.array();
// pixels holds rgb values respectively
// convert the red + green values back into x + y values
pX = (pixels[0] * 255) - 25f;
pY = -((pixels[1] * 255) - 25f);
// draw the proper texture
for ( Entity e : this.entities ) {
e.draw( gl2 );
}
}

You've almost got it. You're going to need a good value for Z in the unproject function though.
What you are trying to do is take the position of the cursor and multiply by a matrix to give a point in "3d space". Your matrices are likely 4x4 or 4x3, so you need a 4 component vector. (x,y,z,w)
When you draw your map, the existing point is multiplied by 1 or more matrices including the projection matrix. ( e.g. -25.0f,25.0f,0.0f,1.0f - actually a 3d point). When this is multiplied by all matrices, the GPU essentially gets back a value in normalised device coordinates (NDC) (between -1 and 1 in all axes) for that vertex.
To do the opposite and unproject you'll need to have a valid/good value for Z. The reason is that in NDC everything that is drawn is in -1,1 on all axes, to get everything in (further away things are squashed a bit). This is how you get flickering and weirdness if you have a huge > 100000 zFar distance for example, it still has to fit into -1,1.
The best way to do this is to use the depth buffer, by capturing the depth value it'll give you a good approxomation of the z coordinate in NDC, which you can pass to the unproject call.
The reason why 0.945 is the sweet spot is probably dependent on how far the camera is from your map or vice versa. It's usually the case that the depth buffer has much more detail closer to the near plane than the far - it's not linear.
http://www.opengl-tutorial.org/beginners-tutorials/tutorial-3-matrices/ has a good visual near the bottom of the page, and is a good resource for intro to matrices in general:
You can see the distortion caused by moving to NDC. This is required for viewing from a perspactive viewpoint, but you need to take it into consideration when you transform backward too.
Colour picking as mentioned is also viable for picking, but will still require some work. Because you have a single object, you'll have to render each texel of the image with a different colour, output that to a separate colour buffer, check to see what colour is on the buffer and somehow relate that to a point in space. It could probably be done though, but I'd say colour picking is more suited to multiple objects.
From what I've read - the depth buffer one might be more suitable for you as it's one object, and the depth buffer will give you a Z coordinate for every point you click on. It could still be on your far plane, but it will still give you a value.
Alternatively, as suggested by #elect use an orthographic projection.

Tracking objects in a 3D environment with 2D HUD icons

I am attempting to have a 2D HUD which has icons that track the location on the screen of 3D objects behind the HUD in a 3D environment.
Reasoning: Sometimes you cannot see a 3D object (too far away or off screen) but you still want to know where it is.
Issue: 3D scene is using a perspective matrix to transform it, giving it depth (z-axis), the HUD is strictly 2D (xy-plane). Because of the depth, the 2D HUD cannot properly track objects when they are farther/closer away.
What I want: A way to get a 2D Vector [(x,y) pos] of where to put an icon so that it is centered where the 3D object in the background would be.
Example of all objects in an xy-plane (z=0):
You can see that as the objects get farther away from the center, the Icon (circle thing in white) is more off center.
Example of objects with increasing depths (farther from center == deeper):
You can see that the HUD thinks 3D objects are in the same plane still.
Pseudo-Code:
.getPos() gets the Vector (x,y,z)
lookAtObj = Object.getPos() - camera.getPos() // lookAt vector from camera to the object
icon.pos = Orthogonal Component of lookAtObj on camera.get_lookAt()
My Perspective Matrix:
// Function call in the OpenGL draw() method
FloatMatrix proj = FloatMatrix.getPerspectiveMatrix( this.fov, this.width, this.height, 0.1f, 200.0f );
// Function
public static FloatMatrix getPerspectiveMatrix( Double fov, float w, float h, float near, float far ){
float asp = w/h;
float fov_cos = (float) Math.cos( fov / 2.0d );
float fov_sin = (float) Math.sin( fov / 2.0d );
float fov_cot = fov_cos/fov_sin;
float a_0 = fov_cot/asp;
float a_3 = (far + near)/(near-far);
float a_43 = (2.0f * far * near)/(near-far);
float[] an = {
a_0, 0.0f, 0.0f, 0.0f,
0.0f, fov_cot, 0.0f, 0.0f,
0.0f, 0.0f, a_3, -1.0f,
0.0f, 0.0f, a_43, 0.0f,
};
return new FloatMatrix( an, 4, 4 );
}

This is pretty straightforward. You can use gluProject. It will take a given modelview, projection, and viewport transform, and a 3D point, and apply the inverse and spit out a 2D point in window coordinates for you (apologies for minor typos, just typing this here):
double myX = ..., myY = ..., myZ = ...; // your object's 3d coordinates
double[] my2DPoint = new double[2]; // will contain 2d window coords when done
double[] modelview = new double[16];
double[] projection = new double[16];
int[] viewport = new int[4];
gl.glGetDoublev(GL2.GL_MODELVIEW_MATRIX, modelview, 0);
gl.glGetDoublev(GL2.GL_PROJECTION_MATRIX, projection, 0);
gl.glGetIntegerv(GL2.GL_VIEWPORT, viewport, 0);
glu.gluProject(myX, myY, myZ, modelview, 0, projection, 0,
viewport, 0, my2DPoint, 0);
// now my2DPoint[0] is window x, and my2DPoint[1] is window y
After you do this, you'll have your 3D point in 2D window coordinates. Then simply switch your projection over to a 2D orthogonal projection, in window pixels, and draw your HUD in 2D space.
For performance, if you have multiple HUD items to draw per frame; just get the modelview/projection/viewport once per frame (or, even better, invalidate your cached ones if you change them and re-query only as needed) and reuse them in subsequent calls to gluProject.

Rotation for rolling 3d ball

I have a Ball in 3D space that has the following attributes:
location - a Vector3f representing where the ball is
rotation - a Vector3f representing the x, y, and z axis rotation angles
I want to roll the ball in a particular direction denoted by a Vector3f "direction". How would I calculate the appropriate axis rotation vector (see above) based on the direction I want the ball to roll towards?
I've tried the following:
set rotation.x to direction.z
set rotation.z to direction.x
Calculate the ball's transform matrix as:
private Matrix4f calculateEntityMatrix(EEntity entity)
{
Matrix4f matrix = new Matrix4f();
matrix.translate(new Vector3f(entity.getXLocation(), entity.getYLocation(), entity.getZLocation()));
if(entity.getXRotation()>0)
{
matrix = matrix.rotate(entity.getXRotation(), new Vector3f(1f, 0f, 0f));
}
if(entity.getYRotation()>0)
{
matrix = matrix.rotate(entity.getYRotation(), new Vector3f(0f, 1f, 0f));
}
if(entity.getZRotation()>0)
{
matrix = matrix.rotate(entity.getZRotation(), new Vector3f(0f, 0f, 1f));
}
if(entity.getXScale()!=1 || entity.getYScale()!=1 || entity.getZScale()!=1)
{
matrix = matrix.scale(new Vector3f(entity.getXScale(), entity.getYScale(), entity.getZScale()));
}
return matrix;
}
This works when rolling down either the x or z axis, but when I roll in a direction between the two axes the rotation appears incorrect. My assumption is that this is caused by the fact that the rotation is being calculated as follows:
the ball is rotated by rotation.x along the X axis
the ball is then rotation by rotation.z along "new" X axis created by step 1.
Any suggestions how this behaviour could be changed so that each rotation is calculated independently of each other?

Unless you want to implement slipping and/or backspin, I think you should approach this problem slightly different. You already have a Matrix.rotate() that supports rotation around an arbitrary axis, use it.
Attributes to store for the ball
position
rotation matrix
Note: in a matrix-oriented system / scene graph, bould would usually be stored in a single 4x4 transformation matrix. That might or might not be more convenience, depending on the rest of your current code base.
Algorithm for moving around
Given the ball direction and the standard up vector (0, 1, 0), calculate the rotation axis using the cross product. (i.e. it's perpendiciular to both direction and up axis)
Rotation is now simply a matrix.rotate( rotationSpeed, rotationAxis), applied to the existing rotation matrix.
If the ball is allowed to roll up/down surfaces, replace the standard up vector with the appropriate surface normal.

3D Picking OpenGL ES 2.0 after model matrix translation

Hey all I'm trying to implement 3D picking into my program, and it works perfectly if I don't move from the origin. It is perfectly accurate. But if I move the model matrix away from the origin (the viewmatrix eye is still at 0,0,0) the picking vectors are still drawn from the original location. It should still be drawing from the view matrix eye (0,0,0) but it isn't. Here's some of my code to see if you can find out why..
Vector3d near = unProject(x, y, 0, mMVPMatrix, this.width, this.height);
Vector3d far = unProject(x, y, 1, mMVPMatrix, this.width, this.height);
Vector3d pickingRay = far.subtract(near);
//pickingRay.z *= -1;
Vector3d normal = new Vector3d(0,0,1);
if (normal.dot(pickingRay) != 0 && pickingRay.z < 0)
{
float t = (-5f-normal.dot(mCamera.eye))/(normal.dot(pickingRay));
pickingRay = mCamera.eye.add(pickingRay.scale(t));
addObject(pickingRay.x, pickingRay.y, pickingRay.z+.5f, Shape.BOX);
//a line for the picking vector for debugging
PrimProperties a = new PrimProperties(); //new prim properties for size and center
Prim result = null;
result = new Line(a, mCamera.eye, far);//new line object for seeing look at vector
result.createVertices();
objects.add(result);
}
public static Vector3d unProject(
float winx, float winy, float winz,
float[] resultantMatrix,
float width, float height)
{
winy = height-winy;
float[] m = new float[16],
in = new float[4],
out = new float[4];
Matrix.invertM(m, 0, resultantMatrix, 0);
in[0] = (winx / width) * 2 - 1;
in[1] = (winy / height) * 2 - 1;
in[2] = 2 * winz - 1;
in[3] = 1;
Matrix.multiplyMV(out, 0, m, 0, in, 0);
if (out[3]==0)
return null;
out[3] = 1/out[3];
return new Vector3d(out[0] * out[3], out[1] * out[3], out[2] * out[3]);
}
Matrix.translateM(mModelMatrix, 0, this.diffX, this.diffY, 0); //i use this to move the model matrix based on pinch zooming stuff.
Any help would be greatly appreciated! Thanks.

I wonder which algorithm you have implemented. Is it a ray casting approach to the problem?
I didn't focus much on the code itself but this looks a way too simple implementation to be a fully operational ray casting solution.
In my humble experience, i would like to suggest you, depending on the complexity of your final project (which I don't know), to adopt a color picking solution.
This solution is usually the most flexible and the easiest to be implemented.
It consist in the rendering of the objects in your scene with unique flat colors (usually you disable lighting as well in your shaders) to a backbuffer...a texture, then you acquire the coordinates of the click (touch) and you read the color of the pixel in that specific coordinates.
Having the color of the pixel and the tables of the colors of the different objects you rendered, makes possible for you to understand what the user clicked from a logical perspective.
There are other approaches to the object picking problem, this is probably universally recognized as the fastest one.
Cheers
Maurizio

How to rotate an object in Java 3D?

I have a Cone I drew in Java 3D with the following code:
Cone cone = new Cone(2f, 3f);
Transform3D t3d = new Transform3D();
TransformGroup coneTransform = new TransformGroup(t3d);
coneTransform.setCapability(TransformGroup.ALLOW_TRANSFORM_WRITE);
t3d.setTranslation(new Vector3f(0f,0f,0f);
coneTransform.setTransform(t3d);
coneTransform.addChild(cone);
this.addChild(coneTransform);
Suppose I have the cone sitting at point (1,1,1) and I want the tip of the cone to point down an imaginary line running through (0,0,0) and (1,1,1)... how can I do this?
Here's an example of what I've been trying:
Transform3D t3d = new Transform3D();
Vector3f direction = new Vector3f(1,2,1);
final double angleX = direction.angle(new Vector3f(1,0,0));
final double angleY = direction.angle(new Vector3f(0,1,0));
final double angleZ = direction.angle(new Vector3f(0,0,1));
t3d.rotX(angleX);
t3d.rotY(angleY);
t3d.rotZ(angleZ);
t3d.setTranslation(direction);
coneTransform.setTransform(t3d);
Thanks in advance for all help!

I'm just learning Java 3D myself at the moment, and from my current knowledge, the rotation methods set the transform to a rotation about that axis only.
Therefore, if you wish to perform rotations about multiple axes, then you will need to use a second Transform3D.
ie:
Transform3D rotation = new Transform3D();
Transform3D temp = new Transform3D();
rotation.rotX(Math.PI/2);
temp.rotZ(Math.PI/2);
rotation.mul(temp); // multiply the 2 transformation matrices together.
As for the reason for Math.PI, this is because it uses radians instead of degrees, where Math.PI is equivalent to 180 degrees.
Finding the angle between your current orientation and your intended orientation isn't too hard - you could use Vector3fs, with the angle() method. A Vector would be set up with the initial orientation, and another in the intended.
However, this doesn't tell you in which axes the angle lies. Doing so would require examination of the vectors to see which segments are set. [of course, there may be something that I am currently unaware of in the API]

This is not a java3D specific answer.
In general a matrix can be built such that there are 4 vectors that describe it.
1) A side (or lateral) vector
2) An up vector
3) A direction vector
4) A position
Each row of a 4x4 matrix.
Thus for a simple identity matrix we have the following matrix (I'll define a column major matrix, for a row major matrix all you need to do is swap the matrix indices around such that row 2 col 3 becomes row 3 col 2 throughout the matrix).
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
in this the first column is the side vector. The second column the up vector. The third the direction and the fourth the position.
Logically we can see that the vector (1, 0, 0, 0) points along the x axis (and thus is the side vector). The vector (0, 1, 0, 0) points along the y axis (and thus is the up vector). The third (0, 0, 1, 0) points along the Z-axis (and thus is the direction vector). The fourth (0, 0, 0, 1) indicates that the objects does not move at all.
Now lets say we wanted to face along the X-axis.
Obviously that would mean we have a vector of (1, 0, 0, 0 ) for our direction vector. Up would still be (0, 1, 0, 0) and position still 0, 0, 0 1. So what would our side vector be? Well, logically it would point along the z-axis. But which way? Well hold your fingers such that one finger points forward, one to the side and one up. Now rotate so that the forward finger is facing the same direction as the side pointing finger. Which way is the side pointing finger pointing now? The opposite direction to the original direction pointing finger. Thus the matrix is
0 0 1 0
0 1 0 0
-1 0 0 0
0 0 0 1
At this point things seemingly get a little more complicated. It is simple enough to take an arbitrary position and an arbitrary point to look at (I'll call them vPos and vFocus). It is easy enough to form a vector from vPos to vFocus by subtracting vPos from vFocus (vFocus.x - vPos.x, vFocus.y - vPos.y, vFocus.z - vPos.z, vFocus.w - vPos.w ). Bear in mind all positions should be defined with a '1' in the w position where all directions should have a '0'. This is automatically taken care of when you do the subtraction above as the 1 in both ws will cancel out and leave 0. Anyway, we now have a vector pointing from the position towards vFocus we'll call it vDir. Unfortunately it has the length of the difference between vPos and vFocus. However if we divide the vDir vector by its length (vDir.x / length, vDir.y / length, vDir.z / length, vDir.w / length) then we normalise it and we have a direction with a total length of 1.
At this ponit we now have our 3rd and 4th columns of our matrix. Now, lets assuem up is still (0, 1, 0, 0) or vUp. We can assume that the crossproduct of the direction and vUp will produce a vector that is perpendicular (and also of unit length) to the plane formed by vDir and vUp. This gives us our side vector or vLat. Now .. we did kind of assume the up vector so its not strictly correct. We can now calculate it exactly by taking the cross product of vLat and vDir and we have all 4 vectors.
The final matrix is thus defined as follows
vLat.x vUp.x vDir.x vPos.x
vLat.y vUp.y vDir.y vPos.y
vLat.z vUp.z vDir.z vPos.z
vLat.w vUp.w vDir.w vPos.w
This isn't strictly the full answer as you will get problems as you look towards a point near to your (0, 1, 0, 0) vector but that should work for most cases.

I finally figured out what I wanted to do by using Quaternions, which I learned about here: http://www.cs.uic.edu/~jbell/Courses/Eng591_F1999/outline_2.html Here's my solution.
Creating the cone:
private void attachCone(float size) {
Cone cone = new Cone(size, size* 2);
// The group for rotation
arrowheadRotationGroup = new TransformGroup();
arrowheadRotationGroup.
setCapability(TransformGroup.ALLOW_TRANSFORM_WRITE);
arrowheadRotationGroup.addChild(cone);
// The group for positioning the cone
arrowheadPositionGroup = new TransformGroup();
arrowheadPositionGroup.
setCapability(TransformGroup.ALLOW_TRANSFORM_WRITE);
arrowheadPositionGroup.addChild(arrowheadRotationGroup);
super.addChild(arrowheadPositionGroup);
}
Now, when I want to rotate the cone to point in a certain direction specified as the vector from the point (0,0,0) to (direction.x, direction.y, direction.z), I use:
private final Vector3f yAxis = new Vector3f(0f, 1f, 0f);
private Vector3f direction;
private void rotateCone() {
// Get the normalized axis perpendicular to the direction
Vector3f axis = new Vector3f();
axis.cross(yAxis, direction);
axis.normalize();
// When the intended direction is a point on the yAxis, rotate on x
if (Float.isNaN(axis.x) && Float.isNaN(axis.y) && Float.isNaN(axis.z))
{
axis.x = 1f;
axis.y = 0f;
axis.z = 0f;
}
// Compute the quaternion transformations
final float angleX = yAxis.angle(direction);
final float a = axis.x * (float) Math.sin(angleX / 2f);
final float b = axis.y * (float) Math.sin(angleX / 2f);
final float c = axis.z * (float) Math.sin(angleX / 2f);
final float d = (float) Math.cos(angleX / 2f);
Transform3D t3d = new Transform3D();
Quat4f quat = new Quat4f(a, b, c, d);
t3d.set(quat);
arrowheadRotationGroup.setTransform(t3d);
Transform3D translateToTarget = new Transform3D();
translateToTarget.setTranslation(this.direction);
arrowheadPositionGroup.setTransform(translateToTarget);
}

I think this should do it:
coneTransform.rotX(Math.PI / 4);
coneTransform.rotY(Math.PI / 4);

you can give your Transform3D a rotation matrix. you can get a rotation matrix using Rotation matrix calculator online: http://toolserver.org/~dschwen/tools/rotationmatrix.html here's my example:
Matrix3f mat = new Matrix3f(0.492403876506104f, 0.586824088833465f,
-0.642787609686539f, 0.413175911166535f, 0.492403876506104f,
0.766044443118978f, 0.766044443118978f, -0.642787609686539f, 0f);
Transform3D trans = new Transform3D();
trans.set(mat);