Stenciling using OpenGl ES 2.0

framebuffer, libgdx, opengl-es, opengl-es-2.0, stencil-buffer

Solution

While drawing to a `SpriteBatch`, state changes are ignored, until `end()` is called. If you want to use stenciling with `SpriteBatch`, you'll need to break up the batch drawing. One thing, I've left out FBOs, but that shouldn't make a difference.

@Override
public void create() {      
    camera = new OrthographicCamera(1, 1);
    batch = new SpriteBatch();

    texture = new Texture(Gdx.files.internal("data/badlogic.jpg"));
    texture.setFilter(TextureFilter.Linear, TextureFilter.Linear);

    TextureRegion region = new TextureRegion(texture, 0, 0, 256, 256);

    sprite = new Sprite(region);
    sprite.setSize(1f, 1f);
    sprite.setPosition(-0.5f, -0.5f);

    spriteUpsideDown = new Sprite(new TextureRegion(texture, 1f, 1f, 0f, 0f));
    spriteUpsideDown.setSize(1f, 1f);
    spriteUpsideDown.setPosition(-0.5f, -0.5f);

    pattern = new Sprite(region);
    pattern.setSize(0.5f, 0.5f);
    pattern.setPosition(-0.25f, -0.25f);

    << Set Input Processor >>
}

The input processor allows to set two boolean flags `breakBatch1` and `breakBatch2` via keyboard (libgdx on desktop), which are used to break the SpriteBatch drawing.

@Override
public void render() {      
    Gdx.gl.glClearColor(1, 1, 1, 1);
    Gdx.gl.glClear(GL10.GL_COLOR_BUFFER_BIT | GL10.GL_STENCIL_BUFFER_BIT);
    batch.setProjectionMatrix(camera.combined);

    // setup drawing to stencil buffer
    Gdx.gl20.glEnable(GL20.GL_STENCIL_TEST);
    Gdx.gl20.glStencilFunc(GL20.GL_ALWAYS, 0x1, 0xffffffff);
    Gdx.gl20.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);
    Gdx.gl20.glColorMask(false, false, false, false);

    // draw base pattern
    batch.begin();
    pattern.draw(batch);

    if(breakBatch1) { batch.end(); batch.begin(); }

    // fix stencil buffer, enable color buffer
    Gdx.gl20.glColorMask(true, true, true, true);
    Gdx.gl20.glStencilOp(GL20.GL_KEEP, GL20.GL_KEEP, GL20.GL_KEEP);

    // draw where pattern has NOT been drawn
    Gdx.gl20.glStencilFunc(GL20.GL_NOTEQUAL, 0x1, 0xff);
    sprite.draw(batch);

    if(breakBatch2) { batch.end(); batch.begin(); }

    // draw where pattern HAS been drawn.
    Gdx.gl20.glStencilFunc(GL20.GL_EQUAL, 0x1, 0xff);
    spriteUpsideDown.draw(batch);
    batch.end();
}

Problem

I am trying to figure out a way to cut out a certain region of a background texture such that a certain custom pattern is not rendered on the screen for that background. For example: This square can be any pattern. I am using Frame Buffer Object and Stencil Buffer to achieve this kind of effect. Here is the code: ``` fbo.begin(); //Disables ColorMask and DepthMask so that all the rendering is done on the Stencil Buffer Gdx.gl20.glColorMask(false, false, false, false); Gdx.gl20.glDepthMask(false); Gdx.gl20.glEnable(GL20.GL_STENCIL_TEST); Gdx.gl20.glStencilFunc(GL20.GL_ALWAYS, 1, 0xFFFFFFFF); Gdx.gl20.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE); stage.getSpriteBatch().begin(); rHeart.draw(stage.getSpriteBatch(), 1); //Draws the required pattern on the stencil buffer //Enables the ColorMask and DepthMask to resume normal rendering Gdx.gl20.glColorMask(true, true, true, true); Gdx.gl20.glDepthMask(true); Gdx.gl20.glStencilFunc(GL20.GL_EQUAL, 1, 0xFFFFFFFF); Gdx.gl20.glStencilOp(GL20.GL_KEEP, GL20.GL_KEEP, GL20.GL_KEEP); background.draw(stage.getSpriteBatch(), 1); //Draws the background such that the background is not rendered on the required pattern, leaving that area black. stage.getSpriteBatch().end(); Gdx.gl20.glDisable(GL20.GL_STENCIL_TEST); fbo.end(); ``` However this is not working at all. How am I supposed to do this using Stencil Buffers? I am also facing some difficulty understanding glStencilFunc and glStencilOp. It would be very helpful if anyone can shed some light on these two. UPDATE: I have also tried producing something of the same kind using glColorMask. Here is the code: ``` Gdx.gl20.glClearColor(0, 0, 0, 0); stage.draw(); FrameBuffer.clearAllFrameBuffers(Gdx.app); fbo1.begin(); Gdx.gl20.glClearColor(0, 0, 0, 0); batch.begin(); rubber.draw(batch, 1); Gdx.gl20.glColorMask(false, false, false, true); coverHeart.draw(batch, 1); Gdx.gl20.glColorMask(true, true, true, false); batch.end(); fbo1.end(); toDrawHeart = new Image(new TextureRegion(fbo1.getColorBufferTexture())); batch.begin(); toDrawHeart.draw(batch, 1); batch.end(); ``` This code is producing this: Instead of something like this: (Ignore the windows sizes and colour tones) Note: I am using the libgdx library.

Original source