How can I render a LibGDX modelInstance on Vuforia AR library? - java

I know most questions on the community should be done with at least some code. But I´m totally lost here, I don´t even know where to start. What I want to do is use the Vuforia AR Library to render LibGDX 3D modelInstances. However, I don't know how can I make Vuforia render the modelInstances or use a libGDX camera as it´s camera.
I've done external research but I have not been able to find useful information. Is there anyone who can help me get started with this?

Please note that I am not versed in Vuforia AR specifically, but the question has gone unanswered for a while, so I will give it a shot.
The Camera in LibGDX is essentially just a wrapper for two 4x4 matrices, the view matrix Camera#view, and the projection matrix Camer#projection (there is another Matrix, the model matrix, used for world space transformations, however I believe [but am not 100% certain] that in LibGDX, this matrix is already incorporated into the view matrix [so Camera#view is actually the model-view matrix]).
Anyway, following on from this, unless there is a simpler solution that I am not aware of, you should be able to use these underlying matrices to deal with projections between the the Vuforia and LibGDX apis.
(recommended further reading: Model, View, Projection matrices in 3D graphics)
Next is rendering LibGDX 3D ModelInstances using Vuforia. A general solution to this problem, would be to convert the ModelInstance into something that Vuforia can recognize. This would be done by taking the mesh/vertex data represented by the LibGDX model, and feeding it directly into Vuforia.
IMO, the best way of going about this, would be to use a a core representation of the model data which can easily be given to both Vuforia and LibGDX (for example, a certain file format that both can recognize, or as raw FloatBuffers which should be easy to wrap up and give to either API). For reference, LibGDX stores models as vertex information in a collection of FloatBuffers, accessible through Model#meshes or ModelInstance#model#meshes.

Ok. So I finfally managed to combine both libraries. I´m not sure if what I´m doing is the most efficient way of working but it has worked for me.
First of all I´m basing myself on the Sample Apps from Vuforia. Specifically using the FrameMarkers example.
I opened an empty LibGDX project, imported the Vuforia jar and copied the SampleApplicationControl, SampleApplicationException, SampleApplicationGLView, SampleApplicationSession, FrameMarkerRenderer and FrameMarker.
Next, I created some attributes on the AndroidLauncher class of LibGDX and initialized all the Vuforia Stuff:
public class AndroidLauncher extends AndroidApplication implements SampleApplicationControl{
private static final String LOGTAG = "FrameMarkers";
// Our OpenGL view:
public SampleApplicationGLView mGlView;
public SampleApplicationSession vuforiaAppSession;
// Our renderer:
public FrameMarkerRenderer mRenderer;
MyGDX gdxRender;
// The textures we will use for rendering:
public Vector<Texture> mTextures;
public RelativeLayout mUILayout;
public Marker dataSet[];
public GestureDetector mGestureDetector;
public LoadingDialogHandler loadingDialogHandler = new LoadingDialogHandler(
this);
// Alert Dialog used to display SDK errors
private AlertDialog mErrorDialog;
boolean mIsDroidDevice = false;
#Override
protected void onCreate (Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
vuforiaAppSession = new SampleApplicationSession(this);
vuforiaAppSession.setmActivity(this);
AndroidApplicationConfiguration config = new AndroidApplicationConfiguration();
// Load any sample specific textures:
mTextures = new Vector<Texture>();
loadTextures();
startLoadingAnimation();
vuforiaAppSession.initAR(ActivityInfo.SCREEN_ORIENTATION_PORTRAIT);
gdxRender = new MyGDX (vuforiaAppSession);
gdxRender.setTextures(mTextures);
initialize(gdxRender, config);
mGestureDetector = new GestureDetector(this, new GestureListener());
mIsDroidDevice = android.os.Build.MODEL.toLowerCase().startsWith(
"droid");
}
I needed to set the activity so I created the setmActivity() on the SampleApplicationSession.
After that I implemented the Libgdx ApplicationAdapter class and passed the vuforiaAppSession as an attribute to access all the stuff I initialized.
public class MyGDX extends ApplicationAdapter {
ModelInstance modelInstanceHouse;
private AnimationController controller;
Matrix4 lastTransformCube;
// Constants:
static private float kLetterScale = 25.0f;
static private float kLetterTranslate = 25.0f;
// OpenGL ES 2.0 specific:
private static final String LOGTAG = "FrameMarkerRenderer";
private int shaderProgramID = 0;
private Vector<com.mygdx.robot.Texture> mTextures;
//SampleApplicationSession vuforiaAppSession;
PerspectiveCamera cam;
ModelBuilder modelBuilder;
Model model;
ModelInstance instance;
ModelBatch modelBatch;
static boolean render;
public SampleApplicationSession vuforiaAppSession;
public MyGDX ( SampleApplicationSession vuforiaAppSession){
super();
this.vuforiaAppSession = vuforiaAppSession;
}
The last important thing to keep in mind is the render() method. I based myself on the the render method of the FrameMarkerRenderer. It has a boolean that gets activated when the camera starts. So I simple changed the variable both on the vuforia AR initialization and the render() method. I had to put the camera on and identity matrix and multiply the model by the modelViewMatrix.
#Override
public void render() {
if (render) {
// Clear color and depth buffer
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT);
// Get the state from Vuforia and mark the beginning of a rendering
// section
State state = Renderer.getInstance().begin();
// Explicitly render the Video Background
Renderer.getInstance().drawVideoBackground();
GLES20.glEnable(GLES20.GL_DEPTH_TEST);
// We must detect if background reflection is active and adjust the
// culling direction.
// If the reflection is active, this means the post matrix has been
// reflected as well,
// therefore standard counter clockwise face culling will result in
// "inside out" models.
GLES20.glEnable(GLES20.GL_CULL_FACE);
GLES20.glCullFace(GLES20.GL_BACK);
cam.update();
modelBatch.begin(cam);
if (Renderer.getInstance().getVideoBackgroundConfig().getReflection() == VIDEO_BACKGROUND_REFLECTION.VIDEO_BACKGROUND_REFLECTION_ON)
GLES20.glFrontFace(GLES20.GL_CW); // Front camera
else
GLES20.glFrontFace(GLES20.GL_CCW); // Back camera
// Set the viewport
int[] viewport = vuforiaAppSession.getViewport();
GLES20.glViewport(viewport[0], viewport[1], viewport[2], viewport[3]);
// Did we find any trackables this frame?
for (int tIdx = 0; tIdx < state.getNumTrackableResults(); tIdx++)
{
// Get the trackable:
TrackableResult trackableResult = state.getTrackableResult(tIdx);
float[] modelViewMatrix = Tool.convertPose2GLMatrix(
trackableResult.getPose()).getData();
// Choose the texture based on the target name:
int textureIndex = 0;
// Check the type of the trackable:
assert (trackableResult.getType() == MarkerTracker.getClassType());
MarkerResult markerResult = (MarkerResult) (trackableResult);
Marker marker = (Marker) markerResult.getTrackable();
textureIndex = marker.getMarkerId();
float[] modelViewProjection = new float[16];
Matrix.translateM(modelViewMatrix, 0, -kLetterTranslate, -kLetterTranslate, 0.f);
Matrix.scaleM(modelViewMatrix, 0, kLetterScale, kLetterScale, kLetterScale);
Matrix.multiplyMM(modelViewProjection, 0, vuforiaAppSession.getProjectionMatrix().getData(), 0, modelViewMatrix, 0);
SampleUtils.checkGLError("FrameMarkers render frame");
cam.view.idt();
cam.projection.idt();
cam.combined.idt();
Matrix4 temp3 = new Matrix4(modelViewProjection);
modelInstanceHouse.transform.set(temp3);
modelInstanceHouse.transform.scale(0.05f, 0.05f, 0.05f);
controller.update(Gdx.graphics.getDeltaTime());
modelBatch.render(modelInstanceHouse);
}
GLES20.glDisable(GLES20.GL_DEPTH_TEST);
modelBatch.end();
}
It´s a lot of code, but I hope it helps the people that try to start integrating both libraries. I don´t think this is efficient but it´s the only solution I´ve come with.

Pablo's answer is pretty good, however, if you are interested in a bit different approach (Calling Vuforia from LibGDX, not the other way) and a more complete example, here is a github repo with a simple 3D model renderer.

Related

AssetManager textures giving black rectangles

I got to that point when I have to replace all new Sprite(texture) with new Sprite(manager.get(imgPath,Texture.class)) in order to optimize sprites creating and do all the loading before starting the game.
The problem is that when I use a common AssetManager to load all textures I get black rectangles.
Still, using a new AssetManager() each time I need to load a Texture works fine (maybe because this manager is quickly garbadge-collected and doesn't have time to dispose the texture it created?) but that way there is no point of using it.
The setupBody() method below is called at the end of my sprite-body wrapper constructor.
the working code (yet useless) ─ one new AssetManager for each sprite:
protected void setupBody(String path){
BodyEditorLoader loader = new BodyEditorLoader(Gdx.files.internal(path));
BodyDef bDef = new BodyDef();
bDef.type = BodyDef.BodyType.DynamicBody;
bDef.position.set(this.position);
bDef.angle = this.rotation;
body = mWorld.createBody(bDef);
body.setUserData(this);
FixtureDef fix = new FixtureDef();
fix.density = 0.1f;
fix.friction = 0f;
fix.restitution = 0.5f;
fix.filter.categoryBits = this.getCategory();
fix.filter.maskBits = this.getMask();
origin = loader.getOrigin("base"+identifier, 1).cpy();
String imgPath = loader.getImagePath("base"+identifier);
AssetManager manager = new AssetManager();
manager.load(imgPath,Texture.class);
manager.finishLoading();
Texture baseSpriteTexture = manager.get(imgPath,Texture.class);
Texture baseSpriteTexture = new Texture(imgPath);
baseSprite = new Sprite(baseSpriteTexture);
loadHighlightedSprite(loader, identifier);
attachFixture(loader, identifier, fix);
}
the code keeping me sleepless ─ the idea is to have a single non-static manager :
protected void setupBody(String path){
/*no changes before */
String imgPath = loader.getImagePath("base"+identifier);
AssetManager manager = SmallIntestineDemoGame.getAssets();
manager.load(imgPath,Texture.class);
/* no changes after */
}
SmallIntestineDemoGame$getAssets() :
public class SmallIntestineDemoGame extends Game {
public AssetManager assets;
#Override
public void create() {
setScreen(new GameScreen());
}
public static AssetManager getAssets() {
return ((SmallIntestineDemoGame) Gdx.app.getApplicationListener()).assets;
}
public void setupAssets(){
this.assets = new AssetManager();
Texture.setAssetManager(assets);
}
}
DEBUGGING :
Debugging my asset manager got me confused. size of values of values of assets map of manager gets stuck at 33 even if I'm adding more:
(source: hostingpics.net)
I tested both cases ─ the app with a new AssetManager() (followed by finishloading and all required stuff) for each sprite creation. AND with a unique asset manager passed to all sprite creations :
I set a breakpoint at draw method and there was almost no difference between the two sprites, except for sprite.texture.glhandle !
When all sprites textures are created using the same manager their textures glhandle is set to 0 and this explains the black rectangles. But when each sprite is created using a locally newly created asset manager, the glhandle has some proper value (2269).
Texture.setAssetManager(assets);
When you set AssetManager then you need to call update() method of AssetManager each time when your game resumed.
If you don't set the AssetManager, the usual managed texture mechanism will kick in, so you don't have to worry about anything.

how to rotate a spatial with physics control in JmonkeyEngine3?

I loaded a 3D model from asset Manager and add CharacterControl as a control. every thing was working correctly but when I tried to rotate the model it did`nt work.
private CharacterControl player;
private Spatial model;
public static final Quaternion YAW045 = new Quaternion().fromAngleAxis(FastMath.PI/4, new Vector3f(0,1,0));
#Override
public void simpleInitApp() {
// add bullet app sate to state manager
bulletAppState = new BulletAppState();
stateManager.attach(bulletAppState);
bulletAppState.getPhysicsSpace().enableDebug(assetManager);
this.addModel();
}
private void addModel(){
model = assetManager.loadModel("Models/Oto/Oto.mesh.j3o");
model.setLocalTranslation(new Vector3f(0,10,0));
capsuleShape = new CapsuleCollisionShape(1f, 7.9f, 1);
player = new CharacterControl(capsuleShape, 1f);
bulletAppState.getPhysicsSpace().add(player);
model.addControl(player);
rootNode.attachChild(model);
model.rotate(YAW045);
}
please help me.
Richard is right.
The CharacterControl class has a setViewDirection() method.
You should really be switching to BetterCharacterControl though as it has better integration. I don't know why CharacterControl is not deprecated.
In general physics objects has a separate "life" as they live in the Bullet PhysicsSpace. For instance the other common physics control: RigidBodyControl class has a setPhysicsRotation(Quaternion rotation) method (just like it has a setPhysicsLocation() method).
More info is in the wiki (although it references CharacterControl) :
Walking Character

(LIBGDX) Rendering Sprite (character) hides a Wall

I'm making a game with Libgdx and I'm using isometric perspective.
I have a problem when rendering my character because the map is loaded from Tiled Map Editor and character is a Sprite.
If I have a wall on layer 0 of the map, when drawing the wall before the Sprite, if the sprite is behind the wall, we will see the Sprite instead of wall (we should see the wall), if rendering the Sprite first, we will see the wall instead of the Sprite when the sprite is in front of the wall (we should see the Sprite). Any idea about fixing this?
I haven't really worked with tiled maps, but I think it goes something like this.
First, in TileEd, create an object layer that will be used for your character sprite. So the layer should be between background stuff and stuff that should obscure your player. Name it something like "characters".
Then, create a MapObject subclass that can render a sprite. Something like this:
public class SpriteMapObject extends MapObject {
private Sprite sprite;
public SpriteMapObject (Sprite sprite) {
this.sprite = sprite;
}
#Override
public Color getColor () {
return sprite.getColor();
}
#Override
public void setColor(Color color){
sprite.setColor(color);
}
public void render(Batch batch){
Color spriteColor = sprite.getColor();
float originalAlpha = spriteColor.a;
spriteColor.a *= getOpacity();
sprite.draw(batch);
spriteColor.a = originalAlpha;
}
}
Now after you load your map and your sprite, you can put the sprite on that layer you prepared:
map.getLayers().get("characters").add(new SpriteMapObject(playerSprite));
You also need to subclass the map renderer so it will render the sprite for you. Use your subclassed version instead of the original. For instance:
public class SpritesOrthogonalTiledMapRenderer extends OrthogonalTiledMapRenderer {
//Override whichever constructor(s) you need
public SpritesOrthogonalTiledMapRenderer (TiledMap map, Batch batch) {
super(map, batch);
}
#Override
public void renderObject(MapObject object) {
super.renderObject(object);
if(object instanceof SpriteMapObject) {
((SpriteMapObject) object).render(batch);
}
}
}
That's the basics of how it works. But if you want your player to be able to be in front or behind of a wall like you described, then you will need to create multiple extra layers, and move your sprite object to different layers when necessary.
Looks to me like Libgdx could use a feature enhancement to OrthogonalTiledMapRenderer so it can draw objects automatically merged into a TileLayer and inserting them into the draw order based on row.
I found an "answer" somewhere else so I share it with you so if any1 else had the problem:
- get players tile location.
- get each tile in that stack ( aka, get a reference to every tile in that cell across each layer.
- check for visibility exceptions on each of those tiles. Store exceptions as properties defined in Tiled. ( will cover properties shortly )
- handle accordingly, most likely by raising or lowering the players render layer for that frame.

Creating a hierarchy for shape classes within a level in Java

I'm working on a Java based game where I need to model some basic shapes. The implementation I have at the moment has a lot of duplicated attributes across shapes of the same dimensions.
Example, I have an Interactive Rectangle which is involved in Box2d world and I have a Background Rectangle which is not involved in Box2d world. But both classes need a width and height defined.
Does anyone know a better way to model this data?
Current implementation ..
Mock idea for new implementation, however Interactive and Background can only inherit from one parent.
The reason why I have Interactive and Background classes is so I can loop through my objects like below.
for (Background bgShape : getLevel.getBGShapes()){
bgShape.draw(
gl2,
new Vec3(
bgShape.getPosition().x,
bgShape.getPosition().y,
bgShape.getPosition().z));
}
for (Body body = world.getBodyList(); body != null; body = body.getNext()) {
Interactive iaShape = (Interactive) body.getUserData();
iaShape.draw(
gl2,
new Vec3(
body.getPosition().x,
body.getPosition().y,
0.0f));
}
Also Interactive class defines a number of attributes which are not present in Background,
public abstract class InteractiveShape extends Shape {
private String bodyType;
private float density;
private float friction;
private float restitution;
private boolean fixedRotation;
private Body body;
}
2nd example is much better asRectangle is really a shape, but has additional data (width and height)
I don't think Iteractive and Background are necessary unless they add any additional operations (which from the picture they do not).

Jmonkey engine 3.0 Drawing points

How can I draw a 3D-point (or point sprite) in 3D space?
There is no documentation for drawing a point in JMonkey Engine site or anywhere else. Just a single point. Then updating the coordinates. No color, just a dot in 3D space.
A point (as opposed to a sphere) can be created using a mesh in which you directly set its buffers (or technically buffer; since a points mesh doesn't require an index buffer as other more complex meshes require. See How can I draw a straight line in the JMonkey Engine library). Mesh creation is documented here.
An example of creating points in 3D space using a mesh is below:
import com.jme3.app.SimpleApplication;
import com.jme3.material.Material;
import com.jme3.math.*;
import com.jme3.renderer.RenderManager;
import com.jme3.scene.*;
import com.jme3.util.BufferUtils;
public class Main extends SimpleApplication {
public static void main(String[] args) {
Main app = new Main();
app.start();
}
#Override
public void simpleInitApp() {
Vector3f[] lineVerticies=new Vector3f[5];
lineVerticies[0]=new Vector3f(2,0,0);
lineVerticies[1]=new Vector3f(-1,0,1);
lineVerticies[2]=new Vector3f(0,1,1);
lineVerticies[3]=new Vector3f(1,1,1);
lineVerticies[4]=new Vector3f(1,4,0);
plotPoints(lineVerticies,ColorRGBA.White);
}
public void plotPoints(Vector3f[] lineVerticies, ColorRGBA pointColor){
Mesh mesh = new Mesh();
mesh.setMode(Mesh.Mode.Points);
mesh.setBuffer(VertexBuffer.Type.Position, 3, BufferUtils.createFloatBuffer(lineVerticies));
mesh.updateBound();
mesh.updateCounts();
Geometry geo=new Geometry("line",mesh);
Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
mat.setColor("Color", pointColor);
geo.setMaterial(mat);
rootNode.attachChild(geo);
}
#Override
public void simpleUpdate(float tpf) {
//TODO: add update code
}
#Override
public void simpleRender(RenderManager rm) {
//TODO: add render code
}
}
This will create the points within pointVerticies as shown below
Later if you need to update infomation in a buffer you can do so using:
VertexBuffer posBuffer = mesh.getBuffer(Type.Position);
posBuffer.updateData(BufferUtils.createFloatBuffer(newData));
posBuffer.setUpdateNeeded();
mesh.updateCounts();
mesh.updateBound();
Or (much more efficiently) you can just attach your geometry to a node and move that (depending on your usage case).
Notes
In its most basic state the Vertex buffer expects x1,y1,z1,x2,y2,z2,x3.... etc with no demarcation between where one vertex ends and the other begins. So the following would enter 3 vertices into the buffer; (1.1,1.2,1.3), (2.1,2.2,2.3) and (3.1,3.2,3.3)
m.setBuffer(VertexBuffer.Type.Position, 3, new float[]{1.1,1.2,1.3,2.1,2.2,2.3,3.1,3.2,3.3});
However the createFloatBuffer() method converts from an array of Vector3f into this form.
Also; its often possible to 'get away with' not calling mesh.updateBound();, however without it objects may be culled because the graphics card believes them to be off screen when actually they are visible

Categories