I am working on some webrtc stuff on Android and trying to figure out how does VideoRendererGui.java works. Unfortunately, I have some trouble with understanding how the following OpenGL code work:
private final String VERTEX_SHADER_STRING =
"varying vec2 interp_tc;\n" +
"attribute vec4 in_pos;\n" +
"attribute vec2 in_tc;\n" +
"\n" +
"void main() {\n" +
" gl_Position = in_pos;\n" +
" interp_tc = in_tc;\n" +
"}\n";
private final String YUV_FRAGMENT_SHADER_STRING =
"precision mediump float;\n" +
"varying vec2 interp_tc;\n" +
"\n" +
"uniform sampler2D y_tex;\n" +
"uniform sampler2D u_tex;\n" +
"uniform sampler2D v_tex;\n" +
"\n" +
"void main() {\n" +
// CSC according to http://www.fourcc.org/fccyvrgb.php
" float y = texture2D(y_tex, interp_tc).r - 15.93;\n" +
" float u = texture2D(u_tex, interp_tc).r - 0.5;\n" +
" float v = texture2D(v_tex, interp_tc).r - 0.5;\n" +
" gl_FragColor = vec4(y + 1.403 * v, " +
" y - 0.344 * u - 0.714 * v, " +
" y + 1.77 * u, 1);\n" +
"}\n";
I was wondering does the above code convert YUV video to RGB. If it does, does it work for all video resolutions?
Here is the link for VideoRendererGui.java
It's using a fragment shader to perform the YUV conversion. The Y, U, and V values are passed into the shader in separate textures, then converted to RGB values for the fragment color. You can see the underlying math on wikipedia.
The shader is sampling the textures, not performing a 1:1 pixel conversion, so any differences in resolution between input and output are handled automatically. The VideoRendererGui code doesn't seem to have any fixed expectations of frame size, so I'd expect it to work for arbitrary resolutions.
Related
I'm trying to add two textures to a 3d cube. I achieved my goal, but on the way I lost the background color.
I want to show the original color of the images and also the background color. I use a mix, but it displays the background completely dark.
This is how it looks my fragmentShaderCode:
private final String fragmentShaderCode =
"precision mediump float;" +
"uniform sampler2D u_Texture0;" +
"uniform sampler2D u_Texture1;" +
"uniform vec4 aColor;" +
"varying vec2 v_TexCoordinate0;" +
"varying vec2 v_TexCoordinate1;" +
"void main() {" +
" vec4 base = texture2D(u_Texture0, v_TexCoordinate0);" +
" vec4 overlay = texture2D(u_Texture1, v_TexCoordinate1);" +
" mediump float ra = (overlay.a) * overlay.r + (1.0 - overlay.a) * base.r;" +
" mediump float ga = (overlay.a) * overlay.g + (1.0 - overlay.a) * base.g;" +
" mediump float ba = (overlay.a) * overlay.b + (1.0 - overlay.a) * base.b;" +
" gl_FragColor = vec4(mix(aColor.rgb, vec4(ra, ga, ba, 1.0).rgb , vec4(ra, ga, ba, 1.0).a), 1.0);" +
"}";
The alpha channel of vec4(ra, ga, ba, 1.0) is 1.0. Therefore the result of vec4(ra, ga, ba, 1.0).a is always 1.0.
You need to use the texture's alpha channels. e.g.: max(base.a, overlay.a):
vec3 textureColor = vec3(ra, ga, ba);
float textureAlpha = max(base.a, overlay.a);
gl_FragColor = vec4(mix(aColor.rgb, textureColor, textureAlpha), 1.0);
Simplify the code by mixing the texture colors with the mix function:
void main() {
vec4 base = texture2D(u_Texture0, v_TexCoordinate0);
vec4 overlay = texture2D(u_Texture1, v_TexCoordinate1);
vec3 textureColor = mix(base.rgb, overlay.rgb, overlay.a);
float textureAlpha = max(base.a, overlay.a);
gl_FragColor = vec4(mix(aColor.rgb, textureColor, textureAlpha), 1.0);
}
I have a ByteBuffer with an RGB frame from a video, so it has size 1280x720x3.
I want to render it using OpenGL. Here's my fragment shader:
public static final String videoFragment =
"#version 320 es\n"+
"precision mediump float;" +
"uniform float alpha;\n" +
"uniform sampler2D sampler;\n"+
"\n" +
"in vec2 TexCoord;\n" +
"out vec4 FragColor;\n" +
"void main()\n" +
"{\n" +
" vec4 rgba;\n" +
" rgba.rgb = texture(sampler, TexCoord).rgb;\n" +
" rgba.a = 1.0f;\n" +
" //FragColor = vec4(1.0f,0,0,1.0f);\n" +
" FragColor = rgba;\n" +
"}\n";
How should I fill it using glTexImage2D? I want to put my RGB data into it. I tried many combinations of GLES20.GL_RGB and others but they won't work or I simply get a black and white image:
GLES20.glTexImage2D(
GLES20.GL_TEXTURE_2D,
0,
GLES20.?,
1280,
720,
0,
GLES20.?,
GLES20.?,
buffer);
Hello I would like to know if there is a way to color a triangle from a mesh without VertexAtribute color. But save it in a seperate array.
EDIT:
Now what I whant is that vertices only has POSITION and no COLOR.
The color of each triangle should be set by a seperate array that holds the color.
I know how to send a uniform to the shader but the render method does render the whole mesh at one and not each triangle.
public class TestBench implements ApplicationListener {
public static final String VERT_SHADER =
"attribute vec2 a_position;\n" +
"attribute vec4 a_color;\n" +
"uniform mat4 u_projTrans;\n" +
"varying vec4 vColor;\n" +
"void main() {\n" +
" vColor = a_color;\n" +
" gl_Position = u_projTrans * vec4(a_position.xy, 0.0, 1.0);\n" +
"}";
public static final String FRAG_SHADER =
"#ifdef GL_ES\n" +
"precision mediump float;\n" +
"#endif\n" +
"uniform vec4 aTest;\n" +
"varying vec4 vColor;\n" +
"void main() {\n" +
" gl_FragColor = vColor;\n" +
"}";
public void create() {
mesh = new Mesh(true, MAX_VERTS, MAX_INDICES,
new VertexAttribute(VertexAttributes.Usage.Position, POSITION_COMPONENTS, "a_position"),
new VertexAttribute(VertexAttributes.Usage.ColorUnpacked, COLOR_COMPONENTS, "a_color"));
}
public void render() {
Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT);
Gdx.gl.glClearColor(0, 0, 0, 1);
flush();
}
void flush() {
mesh.setVertices(vertices);
mesh.setIndices(indices);
Gdx.gl.glDepthMask(false);
Gdx.gl.glEnable(GL20.GL_BLEND);
Gdx.gl.glBlendFunc(GL20.GL_SRC_ALPHA, GL20.GL_ONE_MINUS_SRC_ALPHA);
shader.begin();
shader.setUniformMatrix("u_projTrans", camera.combined);
mesh.render(shader, GL20.GL_TRIANGLES, 0, vertices.lenght);
shader.end();
Gdx.gl.glDepthMask(true);
}
}
If you give a more complete example of your code I'll try to show you how to do it using a uniform in the shaders.
John
I am using a shader I found provided in another stack overflow question to render my screen in grayscale:
import com.badlogic.gdx.graphics.glutils.ShaderProgram;
public class GrayscaleShader {
static String vertexShader = "attribute vec4 a_position;\n" +
"attribute vec4 a_color;\n" +
"attribute vec2 a_texCoord0;\n" +
"\n" +
"uniform mat4 u_projTrans;\n" +
"\n" +
"varying vec4 v_color;\n" +
"varying vec2 v_texCoords;\n" +
"\n" +
"void main() {\n" +
" v_color = a_color;\n" +
" v_texCoords = a_texCoord0;\n" +
" gl_Position = u_projTrans * a_position;\n" +
"}";
static String fragmentShader = "#ifdef GL_ES\n" +
" precision mediump float;\n" +
"#endif\n" +
"\n" +
"varying vec4 v_color;\n" +
"varying vec2 v_texCoords;\n" +
"uniform sampler2D u_texture;\n" +
"\n" +
"void main() {\n" +
" vec4 c = v_color * texture2D(u_texture, v_texCoords);\n" +
" float grey = (c.r + c.g + c.b) / 3.0;\n" +
" gl_FragColor = vec4(grey, grey, grey, c.a);\n" +
"}";
public static ShaderProgram grayscaleShader = new ShaderProgram(vertexShader,
fragmentShader);
}
I would like to be able to fade to grayscale from full color. I'm imagining I should be able to do this by tweening the r,g and b colors which are then set to gl_FragColor (im imagining i will need to create a new ShaderProgram in each setter method used in the tweening process and just use the new color value each time). I understand why (c.r + c.g + c.b)/3.0 is gray, but what can I use to get the values from color to this point? I hope this makes sense!
You can put a uniform float in your fragment shader that blends between the original color and gray like this:
static final String FRAG = "#ifdef GL_ES\n" +
" precision mediump float;\n" +
"#endif\n" +
"\n" +
"varying vec4 v_color;\n" +
"varying vec2 v_texCoords;\n" +
"uniform sampler2D u_texture;\n" +
"uniform float u_grayness;\n" +
"\n" +
"void main() {\n" +
" vec4 c = v_color * texture2D(u_texture, v_texCoords);\n" +
" float grey = (c.r + c.g + c.b) / 3.0;\n" +
" vec3 blendedColor = mix(c.rgb, vec3(grey), u_grayness);\n" +
" gl_FragColor = vec4(blendedColor.rgb, c.a);\n" +
"}";
You can keep a float that tracks the "grayness" from 0 to 1 and apply it like this:
batch.setShader(grayscaleShader);
batch.begin();
grayscaleShader.setUniformf("u_grayness", grayness);
//draw stuff
batch.end();
batch.setShader(null);
By the way, you might have better results by using a proper luminance scale for calculating gray, instead of just summing and dividing by 3. This is because red, green, and blue have different relative luminance as perceived by the eye. Example:
float grey = dot( c.rgb, vec3(0.22, 0.707, 0.071) );
I am using libgdx to make a little platformer and I would like to make the enemies blink in red while the player hurt them with his weapon.
I already tried to change the sprite color and the sprite batch color with no success, it only melt the new color with the one of the texture.
sprite.setColor(Color.RED);
spriteBatch.draw(sprite);
the effect I want to achieve is:
going from sprite texture to full red and then back again.
I think there is something to do with the Blending function, but I am not sure about that. I want to avoid making some red sprite for each monsters of my game.
Does someone know how to achieve this effect?
You can create a shader like this to change the color of all pixels instead of multiplying them by that color. Use this shader with SpriteBatch by calling spriteBatch.setShader(shader);.
This is basically the default SpriteBatch shader, except the final color replaces all non-alpha pixels.
To use this method, you must batch your colored sprites separately from your normal sprites. Call spriteBatch.setShader(null); to go back to drawing regular sprites.
String vertexShader = "attribute vec4 " + ShaderProgram.POSITION_ATTRIBUTE + ";\n" //
+ "attribute vec4 " + ShaderProgram.COLOR_ATTRIBUTE + ";\n" //
+ "attribute vec2 " + ShaderProgram.TEXCOORD_ATTRIBUTE + "0;\n" //
+ "uniform mat4 u_projTrans;\n" //
+ "varying vec4 v_color;\n" //
+ "varying vec2 v_texCoords;\n" //
+ "\n" //
+ "void main()\n" //
+ "{\n" //
+ " v_color = " + ShaderProgram.COLOR_ATTRIBUTE + ";\n" //
+ " v_texCoords = " + ShaderProgram.TEXCOORD_ATTRIBUTE + "0;\n" //
+ " gl_Position = u_projTrans * " + ShaderProgram.POSITION_ATTRIBUTE + ";\n" //
+ "}\n";
String fragmentShader = "#ifdef GL_ES\n" //
+ "#define LOWP lowp\n" //
+ "precision mediump float;\n" //
+ "#else\n" //
+ "#define LOWP \n" //
+ "#endif\n" //
+ "varying LOWP vec4 v_color;\n" //
+ "varying vec2 v_texCoords;\n" //
+ "uniform sampler2D u_texture;\n" //
+ "void main()\n"//
+ "{\n" //
+ " gl_FragColor = v_color * texture2D(u_texture, v_texCoords).a;\n" //
+ "}";
ShaderProgram shader = new ShaderProgram(vertexShader, fragmentShader);