glClear and glStencilMask NOT functioning as expected

c, glut, opengl

Solution

From the docs on glReadPixels()

GL_STENCIL_INDEX 

Stencil values are read from the stencil buffer.
Each index is converted to fixed point, shifted left or right
depending on the value and sign of GL_INDEX_SHIFT,
and added to GL_INDEX_OFFSET. If GL_MAP_STENCIL is GL_TRUE,
indices are replaced by their mappings in the table GL_PIXEL_MAP_S_TO_S.

You might want to check that GL_INDEX_SHIFT and GL_INDEX_OFFSET are both zero by calling

glPixelTransferi(GL_INDEX_SHIFT,  0);
glPixelTransferi(GL_INDEX_OFFSET, 0);

Maybe you're writing the values correctly, but the glReadPixels scrambles the output. Also try reading more than a single byte:

unsigned char pix4[4];
glReadPixels(0,0,1,1,GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, pix4);

for(int i = 0 ; i < 4 ; i ++) printf("Stencil value (0,0,%d): %x\n", i, pix4[i]);

Problem

In Khronos OpenGL 2.1 Specs they say that glStencilMask should affect the operation of glClear, however, on my machine, this does not seem to be true. The output I'm currently getting is: ``` Stencil value: (0,0) 0xff ``` The output I'd expect is: ``` Stencil value: (0,0) 0xf0 ``` Here's my code: ``` void renderScene(void) { unsigned char pix; int i [4]; /* Clear the stencil buffer initially */ glClearStencil(0x0); glClear(GL_STENCIL_BUFFER_BIT); /* Applies the following: 1111 0000 & 1111 1111 */ glStencilMask(0xF0); glClearStencil(0xFF); glClear(GL_STENCIL_BUFFER_BIT); glFlush(); glReadPixels(0,0,1,1,GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, &pix); printf("Stencil value (0,0): %x\n",pix); } int main(int argc, char **argv) { // init GLUT and create Window glutInit(&argc, argv); glutInitDisplayMode(GLUT_DOUBLE | GLUT_RGBA | GLUT_STENCIL ); glutInitWindowPosition(500,500); glutInitWindowSize(320, 320); glutCreateWindow("Trial"); // register callbacks glutDisplayFunc(renderScene); // enter GLUT event processing cycle glutMainLoop(); return 1; } ``` (Obviously I was performing various other drawings etc, but this demonstrates the problem as I see it)

Original source