[BUG] Nested FBO end() corrupts render-surface state, causing incorrect texture Y-flip

[thegreeneyl]

Summary

ofFbo::end() inside another FBO corrupts the matrix stack's render-surface state, causing all subsequent ofTexture::draw() calls within the parent FBO to make wrong Y-flip decisions. Textures rendered after a nested FBO end are vertically flipped.

openFrameworks version

0.12.1 (also present on current master — the code is unchanged)

Platform

All (the affected code is platform-independent C++)

Root cause

In both ofGLProgrammableRenderer::end() and ofGLRenderer::end(), the render surface is unconditionally reset to the window:

// ofGLProgrammableRenderer.cpp, line 1619–1625
void ofGLProgrammableRenderer::end(const ofFbo & fbo) {
    unbind(fbo);
    matrixStack.setRenderSurface(*window);   // ← always resets to window
    uploadMatrices();
    popStyle();
    popView();
}

setRenderSurface(*window) sets currentRenderSurface = nullptr, which changes the return value of customMatrixNeedsFlip():

// ofMatrixStack.cpp, line 117–118
bool ofMatrixStack::customMatrixNeedsFlip() const{
    return vFlipped != (bool(currentRenderSurface) && flipRenderSurfaceMatrix);
}

When inside a parent FBO, currentRenderSurface should point to that parent FBO, making the expression (true && true) = true. After the nested end() resets it to nullptr, the expression becomes (false && true) = false, and customMatrixNeedsFlip() returns the wrong value. This corrupts the orientationMatrix (which is recomputed by setOrientation() during popView()), and downstream, ofTexture::draw() makes the wrong flip decision at:

// ofTexture.cpp, getMeshForSubsection(), line 1103
if (texData.bFlipTexture == vflipped) {
    std::swap(py0, py1);  // flip vertex Y
}

Although popView() does restore vFlipped and orientation from orientationStack, it calls setOrientation() which calls customMatrixNeedsFlip() — and at that point currentRenderSurface has already been clobbered to nullptr.

Steps to reproduce

ofFbo outerFbo, innerFbo;
outerFbo.allocate(512, 512, GL_RGBA);
innerFbo.allocate(512, 512, GL_RGBA);

ofFbo contentFbo;
contentFbo.allocate(512, 512, GL_RGBA);
contentFbo.begin();
// ... draw some content ...
contentFbo.end();

outerFbo.begin();
    // Draw something into a nested FBO (e.g. a blur pass)
    innerFbo.begin();
    ofClear(0);
    contentFbo.draw(0, 0);
    innerFbo.end();

    // Now draw another FBO texture — this will be vertically flipped
    contentFbo.draw(0, 0);  // ← WRONG: vertically flipped
outerFbo.end();

Any real-world scenario with nested FBOs triggers this: blur passes, post-processing chains, multi-layer compositing, etc.

Expected behavior

Drawing contentFbo after innerFbo.end() should produce the same result as drawing it before. The render-surface state should be transparently restored to the parent FBO.

Proposed fix

Add a render-surface stack to ofMatrixStack so that pushView() / popView() save and restore currentRenderSurface and flipRenderSurfaceMatrix, just as they already do for orientation, viewport, and matrices. Then remove the explicit setRenderSurface(*window) from the renderers' end() methods.

ofMatrixStack.h — add to the private section:

struct RenderSurfaceState {
    ofBaseDraws * surface = nullptr;
    bool flipMatrix = false;
};
std::stack<RenderSurfaceState> renderSurfaceStack;

ofMatrixStack.cpp — save in pushView(), restore in popView() (before setOrientation() so customMatrixNeedsFlip() sees the correct surface):

// In pushView(), append:
renderSurfaceStack.push({currentRenderSurface, flipRenderSurfaceMatrix});

// In popView(), insert BEFORE the orientationStack restore:
if(!renderSurfaceStack.empty()){
    auto saved = renderSurfaceStack.top();
    renderSurfaceStack.pop();
    currentRenderSurface = saved.surface;
    flipRenderSurfaceMatrix = saved.flipMatrix;
}

// In clearStacks(), append:
while (!renderSurfaceStack.empty()){
    renderSurfaceStack.pop();
}

ofGLProgrammableRenderer.cpp — remove setRenderSurface(*window), move uploadMatrices() after popView():

void ofGLProgrammableRenderer::end(const ofFbo & fbo) {
    unbind(fbo);
    popStyle();
    popView();
    uploadMatrices();
}

ofGLRenderer.cpp — same change:

void ofGLRenderer::end(const ofFbo & fbo) {
    unbind(fbo);
    popStyle();
    popView();
}

The fix is backward-compatible: without nesting, the stack simply saves and restores the window state, producing identical behavior to stock OF.