I'm currently working on a system which displays video frames (as bitmaps) via OpenGL ES 1.0 on Android. My issue is that I have not been able to get more than about 10 fps.
After doing some testing, I've determined that one of the biggest bottlenecks is the need for the bitmap to have its width and height both be a power of 2. A 640x480 video has to be scaled up to 1024x1024, for example. Without the scaling, I've been able to get about 40-50fps, but the texture just appears white, which does me no good.
I know that OpenGL ES 2.0 supports using non-power of two textures, but I have no experience with shaders / anything else new in 2.0
Is there any way I can get around this issue? How do other video plays get such good performance in comparison to what I have? I have included some code for reference.
private Bitmap makePowerOfTwo(Bitmap bitmap)
{
// If one of the bitmap's resolutions is not a power of two
if(!isPowerOfTwo(bitmap.getWidth()) || !isPowerOfTwo(bitmap.getHeight()))
{
int newWidth = nextPowerOfTwo(bitmap.getWidth());
int newHeight = nextPowerOfTwo(bitmap.getHeight());
// Generate a new bitmap which has the needed padding
return Bitmap.createScaledBitmap(bitmap, newWidth, newHeight, true);
}
else
{
return bitmap;
}
}
private static boolean isPowerOfTwo(int num)
{
// Check if a bitwise and of the number and itself minus one is zero
return (num & (num - 1)) == 0;
}
private static int nextPowerOfTwo(int num)
{
if(num <= 0)
{
return 0;
}
int nextPowerOfTwo = 1;
while(nextPowerOfTwo < num)
{
nextPowerOfTwo <<= 1; // Bitwise shift left one bit
}
return nextPowerOfTwo;
}
Just because a texture has to be a power of two, doesn't mean that your data has to be a power of two.
You're free to create a 1024x1024 (or 1024x512) texture during initialization with glTexImage, fill in the lower 640x480 with your bitmap data with glTexSubImage, and then display the lower 640x480 of the texture with some intelligent texcoords (0,0) to (640/1024, 480/1024). The remainder of the texture will just contain empty space that's never seen.
Related
I love experimenting with operating on images pixel by pixel using pixels[]. However, I’d like to work on a matrix of 100x100 pixels but have the display be several times larger than 100x100 on my screen.
Is there an efficient and straightforward way to zoom, magnify, scale, resize, or etc… to allow me to work with pixels[] just… bigger?
I was disappointed to see that scale() doesn’t work with pixel arrays.
My first idea to scale by a factor of S is by iterating through my pixel array and drawing a board of SxS rects with the fill of each pixel’s color val. However, this is computationally intensive, and I can’t get my frameRate above 5 or so.
Ah great idea #sorifiend. Since I was calling loadPixels() immediately after setting the window size() and leaving it blank (rather than loading pixels from a image and putting it in the window,) I hadn't thought of that. I found you can use createImage() to make an empty image, which resizes crisply with noSmooth(); on! Thanks for the start.
Here's my code for posterity:
void setup() {
i = createImage(100, 100, RGB);
background(0);
size(800, 800);
i.loadPixels();
noSmooth();
}
void draw() {
i.loadPixels();
for (int x = 0; x < i.width; x++) {
for (int y = 0; y < i.height; y++) {
if (random(0,max(i.width,i.height)) < y) {
i.pixels[x + y*i.height] = color(255);
} else {
i.pixels[x + y*i.height] = color(0);
}
}
}
i.updatePixels();
image(i,0,0,width,height);
}
I would like to build a kind of image morphing tool in Processing. Similar to what you can see in this link:
https://giphy.com/gifs/painting-morph-oil-c8ygOpL64UDuw
My first step to achieve this was to build a two-dimensional grid of pixels. The pixels are filled with colour. The fill colour is created by reading colour from an image (PImage img1;) with the get(); function. This is how I recreated an image with my pixels. In the second step, I thought I would use the lerp(); function to give the respective pixels the colour of a second image (PImage img2;) - I thought this would create the desired morph effect. But I was wrong! The whole thing works - but the effect is only that a fade-in takes place between the two images. And no morphing. What exactly happens to pixels while this morph effect? How could I recreate it in Processing?
float pixel;
float pixelsize;
PImage img1;
PImage img2;
float counter;
void setup() {
size(1080, 1080);
pixel = 100;
pixelsize = width/pixel;
noStroke();
img1 = loadImage("0.jpg");
img2 = loadImage("1.jpg");
counter = 0;
}
void draw() {
background(255);
for (int y = 0; y < pixel; y++) {
for (int x = 0; x < pixel; x++) {
color c1 = img1.get(int(pixelsize*x), int(pixelsize*y));
color c2 = img2.get(int(pixelsize*x), int(pixelsize*y));
color from = c1;
color to = c2;
color interA = lerpColor(from, to, counter);
pushMatrix();
translate(pixelsize*x, pixelsize*y);
fill(interA);
rect(0, 0, pixelsize, pixelsize);
popMatrix();
}
}
counter= counter + 0.01;
}
Indeed it is not a straight forward task.
You're approach is not a bad start: it would result in a nice crossfade between the two images.
Bare in mind get() can be costly on the CPU.
You can however use the pixels[]:
PImage img1;
PImage img2;
// transition image
PImage imgT;
void setup() {
size(1080, 1080);
img1 = loadImage("0.jpg");
img2 = loadImage("1.jpg");
// copy the 1st image (copies width/height as well)
imgT = img1.get();
}
void draw() {
background(255);
// map transition amount to mouse X position
float t = map(mouseX, 0, width, 0.0, 1.0);
// make all pixels readable
imgT.loadPixels();
// lerp each pixel
for(int i = 0 ; i < imgT.pixels.length; i++){
imgT.pixels[i] = lerpColor(img1.pixels[i], img2.pixels[i], t);
}
// update all pixels in one go
imgT.updatePixels();
// display result
image(imgT, 0, 0);
}
Implementing a full morph image is non-trivial.
I can recomend two options to make use of existing algorithms, however these options are also not beginner friendly:
ImageMagick implements shepards distortion and there is a java library that interfaces with imagemagick: im4java. Note that you'd need to download the precompiled java library and drop the .jar file on top of your sketch and processing the output might take time: probably not feasible for realtime (however it should be possible to save individual frames to disk and assemble them as a gif/movie/etc.)
Using OpenCV: there's an OpenCV Face Morph tutorial with source code in c++ or Python and there is a Processing OpenCV library. It would be a matter of porting the c++/Python OpenCV calls to the Java OpenCV API.
In my app i'm iterating through URLs of images, decoding and putting them in an ArrayList<Bitmap>.
They may vary greatly in size, so i'm doing a "pre-decode" with the inJustDecodeBounds = true option to calculate the necessary inSampleSize value for the actual decode.
See my method for this below, i hope it's not too hard to understand. Basically i'm aiming for a size similar to the screen size of the device.
for (Element e: posts) {
if (!e.id().equals("")) {
//preparing decode
options = new BitmapFactory.Options();
input = new URL(e.url).openStream();
options.inJustDecodeBounds = true;
BitmapFactory.decodeStream(input, null, options);
input.close();
//setting inSampleSize if necessary
int picPixels = options.outWidth * options.outHeight;
int picScreenRatio = picPixels / screenPixels;
if (picScreenRatio > 1) {
int sampleSize = picScreenRatio % 2 == 0 ? picScreenRatio : picScreenRatio + 1;
options.inSampleSize = sampleSize;
}
//actual decode
input = new URL(e.url).openStream();
options.inJustDecodeBounds = false;
Bitmap pic = BitmapFactory.decodeStream(input, null, options);
input.close();
picList.add(pic);
}
}
Code for calculating screenPixels:
Display display = getWindowManager().getDefaultDisplay();
Point screenSize = new Point();
display.getSize(screenSize);
int screenPixels = screenSize.x * screenSize.y;
I'm going through ~60 images and around 40 my app crashes with java.lang.OutOfMemoryError.
As i understand, with inJustDecodeBounds = true there's no memory allocated, and if my method is correct (i believe it is), very big images get very big inSampleSizes, so i don't know what could be the problem.
Would appreciate any advice.
No matter that you use inSampleSize you can't easily hold 60 bitmaps as large as your screen in memory.
Assuming a fullhd screen you get about 1920*1080*60*4 bytes.
Thats roughly 500 MB of data.
You need to somehow change your bitmap loading logic.
In this answer you can find out how much memory you can count with. It differs for various devices.
But always try to make you app as memory efficient as possible.
Sample size must be a power of two. From the documentation:
If set to a value > 1, requests the decoder to subsample the original
image, returning a smaller image to save memory. The sample size is
the number of pixels in either dimension that correspond to a single
pixel in the decoded bitmap. For example, inSampleSize == 4 returns an
image that is 1/4 the width/height of the original, and 1/16 the
number of pixels. Any value <= 1 is treated the same as 1. Note: the
decoder uses a final value based on powers of 2, any other value will
be rounded down to the nearest power of 2.
Here's a good trick from the Android - Volley library on how to find the best sample size for bitmaps:
int findBestSampleSize(int actualWidth, int actualHeight, int desiredWidth, int desiredHeight) {
double wr = (double) actualWidth / desiredWidth;
double hr = (double) actualHeight / desiredHeight;
double ratio = Math.min(wr, hr);
float n = 1.0f;
while ((n * 2) <= ratio) {
n *= 2;
}
return (int) n;
}
It is notoriously hard to get image manipulation right on Android. I would recommend that you use an established Image Management Library like Facebook's Fresco:
https://github.com/facebook/fresco
An example from their docs:
Fresco's Drawees show a placeholder for you until the image has loaded and automatically show to the image when it arrives. When the image goes off-screen, it automatically releases its memory.
Also:
Apps using Fresco can run even on low-end devices without having to constantly struggle to keep their image memory footprint under control.
I want to shade a bitmap from one color to the other gradually (So as if it's going to glow from not so red to gradually getting redder.
I'm doing this per pixel but it's causing the app to jump, can someone show me a more efficient method?
private void adjustCountryBitmapColor()
{
//TODO Possible memory leak in this method.
mAllPixels = new int [ mCountryBitmap.getHeight()* mCountryBitmap.getWidth()];
mCountryBitmap.getPixels(mAllPixels, 0, mCountryBitmap.getWidth(), 0, 0, mCountryBitmap.getWidth(),mCountryBitmap.getHeight());
/*
int alpha=argb>>24;
int red=(argb & 0x00FF0000)>>16;
int green=(argb & 0x0000FF00)>>8;
int blue=(argb & 0x000000FF);
*/
for(int i =0; i< mCountryBitmap.getHeight()*mCountryBitmap.getWidth(); i++)
{
if(mAllPixels[i]>>24 == -1)
{
/*AllPixels[i] == Color.BLACK)
{
mAllPixels[i] = Color.RED;
}
*/
mAllPixels[i] = Color.RED;
}
}
System.out.println(mDeltaOffSet);
//int alpha = mAllPixels[0]>>24;
//System.out.println(alpha);
mCountryBitmap.setPixels(mAllPixels, 0, mCountryBitmap.getWidth(), 0, 0, mCountryBitmap.getWidth(), mCountryBitmap.getHeight());
}
Don't create an intermediate bitmap, that's going to take too long; there's Canvas#drawBitmap(int[],....) for exactly that reason. Also, have look whether Paint#setColorFilter() with a ColorMatrixFilter and a ColorMatrix can cover your color transforms.
I had found a good Image Processing library which is using GPU. Its is so fast that you can do live camera as well immediate image processing.
Library is provided by javierpuntonet / android-gpuimage. Just take a look at it and I am sure, you will get benefit of using it.
So I've got this nice Android game (a snake-clone with animations), doing the final testing, when BAM! My second testing device (Nexus 1, HTC Magic was my 1.) flickers when drawing.
Does anyone know why this code won't work correctly with the Nexus 1?
public void draw(Canvas canv) {
int count = 0;
isHead = false;
for (int i = 0; i < SPACES; i++) {
if (mDrawSpaces[i]) {
count++;
if (count == SPACES - 1) {
setDrawSpacesToFalse();
if (bmp != null)
super.drawPlaceable(canv);
}
} else {
mDrawSpaces[i] = true;
return;
}
}
}
I have a list of Birds (Birds / UFOs / others) with SPACES (4) times as many elements which are being drawn on the screen. So I thought to myself, instead of calculating the rotation and scale of the pictures for every Bird, I merely have 3 placeholders between the Birds which each have a picture to be drawn once they're set to visible. These pictures are generated by the first Bird:
public void drawHead(Canvas canv) {
//calculate the rotation & mirroring of the picture
super.drawPlaceable(canv);
//generate the pics for smaller birds following it
mat.preScale((float) 0.6, (float) 0.6);
this.bmp = Bitmap.createBitmap(SPRITESHEET, Bird.mCurFrame
* BIG_W[mUseBird], 0, BIG_W[mUseBird], BIG_H[mUseBird],
mat, true);
}
Any ideas? Is my draw(Canvas) method wrong in some part?
EDIT: I don't know why, I don't know how, but this afternoon when I tested it again, it magically worked...
I can see you are using matrix to scale - another option would be to use
canvas.DrawBitmap(spriteSheet, fromRect, toRect, paint);
Where toRect should be a Rect class of any size, in this way you would create no bitmap objects when drawing game frames. The piant should have filter bitmap enabled.
To rotate you would have to use:
canvas.save();
canvas.rotate(spriteAngle,spriteCenterX, spriteCenterY);
canvas.DrawBitmap(spriteSheet, fromRect, toRect, paint);
canvas.restore();
This is a fast enough code for many 2D games, though not as fast and powerful as OpenGL.