Chaosgoo Do You Find IT?
Go back

Integrating lib3mf, Part 4: Flutter Texture and a Native C++ OpenGL Rendering Pipeline

Published:  at  12:56 AM
⏱️ 1407 words • 8 min read

阅读中文版

Connecting Flutter Texture to Android EGL and native C++ through JNI to build an OpenGL rendering pipeline without adding a PlatformView hierarchy.

Preface

In the previous article in this series, we successfully parsed the 3MF model data at the bottom of C++, but we have not yet rendered and displayed it on the page. In this article, we will introduce how to use the parsed data to render a 3D model through OpenGL.

This article may require you to read a previous articleFlutter embeds Android native View and interacts with native. In that article, we successfully embedded ordinary Android controls through PlatformView, so we naturally planned to repeat the same trick and directly embed a GLSurfaceView for 3D rendering. But we directly hit the “south wall” in our actual attempt:

The essence of GLSurfaceView is to forcibly “dig a bottom-out hole” in Android’s native View tree, and its Surface enjoys an independent Z-Order level. This has a serious philosophical conflict with Flutter’s single-layer self-drawn rendering pipeline based on the engine (Skia/Impeller). If you forcibly insert an extremely high-frequency self-redrawing EGL surface into PlatformView (even using Hybrid-Composition), you will face: The layering is disordered, the black screen flickers, and the frame synchronization at both ends (Android/Flutter) is completely torn, so that there is a serious sense of fragmentation when sliding.

Therefore, in order to obtain truly painless native performance, we chose to “restart from scratch” and completely switched to Flutter’s official most recommended high-frequency rendering solution in video/camera stream processing - External textures (Texture component). Combining the native-side SurfaceTexture and the fully manually managed EGL context with the background frame rendering thread, we can deliver the underlying calculated 3D images directly to Flutter’s self-drawing pipeline without “digging holes” or any extra layer overhead. The core implementation involved in this article is based on the reconstructed solution model_file_picker_route.dart below (see the Gist link at the end of the article).

OpenGL decentralization idea (Flutter -> Android -> JNI -> C++)

Our core idea is to pass external textures (Texture Widget) in Flutter to the Android native layer. In the native layer, we will not directly create a new View with a life cycle, but rely on TextureRegistry to apply for a texture, and use HandlerThread and EGL to manage the context by ourselves. Subsequently, the native EGL thread will forward OpenGL life cycle events (creation, size changes, frame-by-frame rendering) to the underlying C++ through JNI. The complete steps are as follows:

  1. Flutter applies for textures and obtains the ID: Use MethodChannel to evoke the Android native (Kotlin) environment to create textures.
  2. Android natively creates SurfaceTexture with EGL context: is registered to obtain SurfaceTexture, based on which EGLSurface is created and makeCurrent is created in the background rendering thread.
  3. Based on Choreographer synchronous refresh: replaces GLSurfaceView’s own rendering management, uses the system’s Choreographer to obtain the VSYNC signal, and drives the underlying onDrawFrame.
  4. JNI throws the declaration cycle to C++: Android calls the C++ layer exposed NativeRenderBridge through NativeDrawerRenderer, and the bottom layer SceneRenderer completes the drawing.

Flutter side implementation: Texture Widget and MethodChannel bridging

In model_file_picker_route.dart, we need to get a native Texture ID and embed it into the component tree via that ID:

// 请求 Android 原生层分配 Texture
Future<void> _setupNativeTexture() async {
  final id = await MethodChannel(
    "com.chaosgoo.metasequoia/gl_texture2",
  ).invokeMethod('createGLTexture', {'width': 512, 'height': 512});
  setState(() {
    _textureId = id;
  });
}

Next, in the interface construction part, as long as _textureId is ready, we can use Texture Widget to mount directly:

// 借助 LayoutBuilder 侦测外部约束变化
LayoutBuilder(
  builder: (context, constraints) {
    final double pixelRatio = MediaQuery.of(context).devicePixelRatio;
    final double width = constraints.maxWidth * pixelRatio;
    final double height = constraints.maxHeight * pixelRatio;
    _updateNativeTexture(width.toInt(), height.toInt());

    return GestureDetector(
      onPanUpdate: (details) {
        const sensitivity = 0.01;
        MethodChannel("com.chaosgoo.metasequoia/gl_texture2")
            .invokeMethod('rotate', {
                'textureId': _textureId,
                'angleX': details.delta.dx * sensitivity,
                'angleY': details.delta.dy * sensitivity,
        });
      },
      child: Texture(textureId: _textureId!),
    );
  },
)

Through GestureDetector and the gesture capture function onPanUpdate, the touch offset can also be passed to the MethodChannel in the form of axis rotation.

Android side: textures management and EGL background thread

Entering the Android layer, we need to accept Flutter requests and open up our own OpenGL rendering environment.

First, we accept requests and route processing in the GLTexturePlugin2 plugin:

"createGLTexture" -> {
    val handler = GLTextureHandler2(textureRegistry, emptyMap())
    val width = call.argument<Int>("width") ?: 640
    val height = call.argument<Int>("height") ?: 640
    val textureId = handler.getTextureId()
    textureHandlers[textureId] = handler

    handler.setup(width, height)
    result.success(textureId)
}

Since we are separated from GLSurfaceView, we must build an independent HandlerThread and EGL environment in GLTextureHandler2. Once established, use the system’s VSync to obtain the frame refresh timing (via Choreographer.getInstance()).

// Choreographer.FrameCallback
private val frameCallback = object : Choreographer.FrameCallback {
    override fun doFrame(frameTimeNanos: Long) {
        if (!isRendering) return
        renderThreadHandler.sendEmptyMessage(DRAW)
        choreographer.postFrameCallback(this)  // 继续监听下一帧
    }
}

// 核心 Handler 处理渲染生命周期逻辑
val cb = Handler.Callback { msg ->
    when (msg.what) {
        INIT -> {
            Surface(surfaceTexture).apply {
                surface = this
                eglCore = EglCore(this).apply { makeCurrent() }
                render = NativeDrawerRenderer()
                render?.onSurfaceCreated(null, null)
                val (width, height) = msg.obj as? Pair<Int, Int> ?: (0 to 0)
                render?.onSurfaceChanged(null, width, height)
            }
            start() // 开始监听 Choreographer VSYNC
        }
        DRAW -> {
            render?.onDrawFrame(null)
            eglCore?.swapBuffer() // 更新到屏幕/Texture中
        }
        LOAD_MODEL -> {
            val modelPath = msg.obj as? String ?: ""
            (render as? NativeDrawerRenderer)?.loadModel(modelPath)
        }
        // ... 其他命令 (UPDATE_SIZE, ROTATE 等)
    }
    return@Callback true
}

This design throws a lot of calculation and rendering logic to the background thread (RenderThread), and because we initially bound Flutter’s SurfaceTexture, when eglCore?.swapBuffer() is triggered, Flutter will automatically listen to the data changes and complete the rendering on the screen through the internal stream.

JNI writing: communicating with the underlying OpenGL

At the Android Kotlin level (NativeDrawerRenderer.kt), it’s just a simple wrapper Wrapper down to C++:

class NativeDrawerRenderer() : GLSurfaceView.Renderer {
    override fun onDrawFrame(gl: GL10?) {
        NativeRenderBridge.onDrawFrame(gl)
    }
    //... 其他事件的封装
}

Since we need to connect to the underlying C++ and call 3MF parsing and various OpenGL instructions, we will directly enter the C++ layer implementation through NativeRenderBridge.java. For example, the JNI side (native_renderer_bridge.cpp) takes over the model life cycle:

static SceneRenderer *g_renderer = nullptr;

extern "C" JNIEXPORT void JNICALL
Java_com_example_native_1lib3mf_NativeRenderBridge_onSurfaceCreated(
        JNIEnv *env, jclass clazz, jobject gl, jobject config) {
    if (g_renderer) {
        delete g_renderer;
    }
    g_renderer = new SceneRenderer();
    g_renderer->init(getAssetManager());
}

extern "C" JNIEXPORT void JNICALL
Java_com_example_native_1lib3mf_NativeRenderBridge_onDrawFrame(JNIEnv *env,
                                                               jclass thiz,
                                                               jobject gl) {
    if (g_renderer) {
        g_renderer->render();
    }
}

// 对应我们在模型选择路由中的模型加载事件
extern "C" JNIEXPORT void JNICALL
Java_com_example_native_1lib3mf_NativeRenderBridge_loadModel(JNIEnv *env, jclass clazz,
                                                             jstring path) {
    const char *nativePath = env->GetStringUTFChars(path, nullptr);
    if (g_renderer) {
        g_renderer->loadModel(nativePath);
    }
    env->ReleaseStringUTFChars(path, nativePath);
}

First flight in stages: screen clearing and basic triangle test

In order to verify whether the Flutter -> Android -> C++ link we built has actually been opened alive, we can write the most primitive OpenGL ES “Hello Triangle” or screen-clearing test code in the onDrawFrame hook at the bottom of C++ without rushing to process the huge 3MF data:

extern "C" JNIEXPORT void JNICALL
Java_com_example_native_1lib3mf_NativeRenderBridge_onDrawFrame(JNIEnv *env,
                                                               jclass thiz,
                                                               jobject gl) {
    // 1. 最基础的清屏测试：将背景清空为红色，验证 EGL 上下文与 VSync 是通畅的
    glClearColor(1.0f, 0.1f, 0.1f, 1.0f);
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    // 2. 基础三角形测试（可选步骤）
    // 你可以在此地硬编码三个顶点坐标（如 0.0, 0.5, 0.0），并加载极其简单的无矩阵 Shader
    // 使用 glDrawArrays(GL_TRIANGLES, 0, 3) 画出一个占据屏幕中心的基础三角形。

    // 如果 g_renderer 已就绪，也可以将上述写死测试的代码封装在里面
    if (g_renderer) {
        // g_renderer->renderTestTriangle();
        g_renderer->render();
    }
}

As soon as you run the code and call the route in Flutter, if you can see that the Texture area of the component tree that should be blank steadily shows the dark red background dyed by the underlying C++, or a native triangle drawn without relying on any Flutter Canvas API appears in the center - then congratulations, this bridge connecting the cross-end UI and the underlying GPU graphics card has been completely completed.

Conclusion

By using Flutter’s built-in Texture component, we have perfectly stripped away the unnecessary overhead and layering issues caused by PlatformView / GLSurfaceView in the Android platform, and completely independently controlled the underlying OpenGL EGL context in the background thread. Using JNI to map the rendering life cycle and events frame by frame to the C++ layer to encapsulate the rendering engine, our next focus can be on how to complete the real Shader development at the C++ level and accurately render and display the vertex data in the 3MF model file.

Appendix

gist link

Share this post on:
Share this post via WhatsAppShare this post on FacebookShare this post on XShare this post via TelegramShare this post on PinterestShare this post via Mail

Continue reading

Previous Post
Integrating lib3mf, Part 5: Extracting 3MF Geometry in C++ and Rendering with OpenGL
Next Post
Integrating lib3mf, Part 3: Reading 3MF File Information in Flutter