Embedding Files in ESP-IDF Firmware with embed_txtfiles

0x00 background

Recently, when I was looking through the search keywords of GSC (Google Search Console), I found that the frequency of the word esp-idf embed-txtfiles was surprisingly high. It seems that when everyone was developing ESP32, they were all troubled by the problem of “how to elegantly place a file in the firmware”.

Before this, in order to implement some functions (such as the previous That desktop pixel small screen), I once silly converted all the pictures into a huge C array (.h file). The results can be imagined: the code file expanded to tens of thousands of lines, the IDE index was stuck, and it took a long time to recompile every time a pixel was fine-tuned. It was simply a disaster.

Whether it is the root certificate of HTTPS, the front-end HTML page, or a simple configuration file, if they are all written in huge const char[] and hard-coded into the code, then engineering maintenance will be a self-torture. Today I will take my piece of which I bought with a huge sum of 9.9 yuan ESP32-C3 and demonstrate how to use it ESP-IDF This black technology in it elegantly fills the pit.

0x01 Plan comparison

Why is embed_txtfiles recommended? We can first look at an intuitive comparison table:

0x02 Get started quickly

The ESP-IDF build system (based on CMake) allows you to take binary or text files directly on disk and automatically convert and “stitch” them into your .bin during the compilation phase. in firmware.

0x00 Prepare resource files

Create a folder (e.g. assets) in your project or component directory and put your text file (e.g. hello.txt) in it.

0x01 Modify CMakeLists.txt

In CMakeLists.txt in the component directory, you only need to specify the path through the EMBED_TXTFILES parameter in the idf_component_register function:

## main/CMakeLists.txt
idf_component_register(SRCS "main.c"
                    INCLUDE_DIRS "."
                    EMBED_TXTFILES "assets/hello.txt")

0x02 Called in C/C++ code

The compilation system automatically generates three global symbols. Their naming convention is: _binary_[filename]_[extension]_start and _binary_[filename]_[extension]_end.

##include <stdio.h>
##include "esp_log.h"

// 0x01. 声明外部资源变量
extern const uint8_t hello_txt_start[] asm("_binary_hello_txt_start");
extern const uint8_t hello_txt_end[]   asm("_binary_hello_txt_end");

void app_main(void)
{
    // 0x02. 获取资源长度
    size_t size = hello_txt_end - hello_txt_start;

    // 因为是 txtfiles，所以末尾自带 \0，可以直接作为字符串处理
    ESP_LOGI("APP", "Embedded file content: %s", (char *)hello_txt_start);
    ESP_LOGI("APP", "File Size: %d bytes", size);
}

Tips: If you are using the .cpp file, please be sure to add extern "C" to the statement, otherwise it will cause Name Mangling Causes the link to not find the symbol.

0x03 Filling the pit: Those “rollover” moments that made me collapse

Although this trick is fun to use, those who are new to it will most likely jump repeatedly in several inexplicable pits like I did back then. Here are a few “rollover” scenes that I personally experienced:

0x00 Rollover scene: Symbol not found

This is the most disturbing error message. Usually it’s because I foolishly wrote an absolute path, or the path was calculated incorrectly.

• Lessons learned from blood and tears: The path in EMBED_TXTFILES must be relative to the current CMakeLists.txt. If you embed a file in the main directory but want to call it in other custom components, the linker will teach you how to do it in minutes. My solution at the time was: Which component needs to be used, just stuff the resources into the folder of that component.

0x01 Modify the read-only memory and explode directly

Since EMBED_TXTFILES thoughtfully added \0 for us, the first time I used it, I actually wanted to use strtok to cut this text. The result? ESP32 directly exploded (LoadProhibited), and kept restarting in a loop.

• Filling summary: Don’t forget that this thing exists in Flash, it is read-only! If you want to modify it, you must first use malloc to open a battlefield, and then use memcpy to import the data there.

0x02 The “metaphysics” of variable naming

All the minus signs and periods in the file names are changed to underlines. Sometimes the rules are so complicated that I doubt my life. For example, if my file is named my-config.v1.json, the generated symbol name will become _binary_my_config_v1_json_start.

• Verification of Dafa: Stop guessing, it’s too dangerous. Go directly to the build directory and launch nm Dafa to view the generated symbols.

0x04 Advanced: How did I verify these “ghost symbols”

0x00 Don’t guess, just look at the product

As a hard-core (hard-core) developer, when I am really in doubt, I will directly look at the generated .elf file. Running this command in your project directory is much more effective than searching for documents for a long time:

nm -gC build/*.elf | grep _binary

If you can see the expected start/end address in the list, then your mentality will be stable.

0x01 Practical combat: “Minimalism” in certificate processing

When tossing about HTTPS or MQTTS, certificate processing is a disaster. I used to write file system drivers honestly, but now I do this directly in the code:

// main.c
// 这里的 asm 名称就是通过上面 nm 命令确定的
extern const uint8_t mqtt_ca_pem_start[] asm("_binary_mqtt_ca_pem_start");

esp_mqtt_client_config_t mqtt_cfg = {
    .broker.address.uri = "mqtts://your-iot-server.com",
    .broker.verification.certificate = (const char *)mqtt_ca_pem_start,
};

Throw the starting pointer directly in, without even writing the length (because TXTFILES has its own terminator). This feeling can only be described as “silky”.

0x02 EMBED_FILES vs EMBED_TXTFILES: This is where I fell.

This is the easiest place for many novices to “step into pitfalls”, and I fell into this trap back then. ESP-IDF provides two similar parameters, the difference is crucial:

• EMBED_FILES: Pure binary embedding. It stuffs the files into the firmware intact. Suitable for non-text resources such as images, fonts, and compressed packages.
• EMBED_TXTFILES: Specifically for text. It will automatically append a null terminator \0at the end of the embedded resource.

If you mistakenly use EMBED_FILES to embed the certificate, you will find that the code will inexplicably read garbage data behind the memory at the end of the run, causing the TLS verification to fail. Practical advice: HTTPS certificate (PEM format) or HTML Please be sure to use EMBED_TXTFILES.

0x03 The “magic” of naming rules

How is the name generated? The rule is simple: all non-alphanumeric characters in the filename (e.g. ., -) will be replaced with underscores _.

For example:

• File path: certs/ca.pem -> Variable name: _binary_ca_pem_start
• When declared in C language: asm("_binary_ca_pem_start")

_Tip: It is recommended to always use relative paths to CMakeLists.txt, otherwise variable names may become very long and unpredictable. _

Why not just use the file system?

Although SPIFFS or LittleFS are also very convenient, embed_txtfiles is the optimal solution in the following scenarios:

1. Security: The certificate is placed in the firmware and is not easily lost due to file system damage. OTA updating the firmware also updates the certificate.
2. Minimalist solution: Avoid tossing partition table mirroring just for a few hundred bytes.
3. Read speed: Data is mapped directly to the instruction bus through the MMU, and the read speed is as fast as reading code, without the addressing overhead of the file system.

0x05 finally

Converting from the clunky C array to the elegant embed_txtfiles essentially returns the resource to its original format and leaves the hard work of generation to the build system. As I felt before in Claude Code first experience, the charm of modern development lies in the constant use of tools to untie those repetitive and inefficient knots.

This time I will share my experience in filling in the pits. I hope it can help you who are tortured by resource paths. See you at the next pit, call it a day!

environment

Framework: ESP-IDF v5.x
Target: ESP32 / S2 / S3 / C3 (RISC-V)
Language: C / C++
Build System: CMake

Reference

• ESP-IDF Programming Guide - Guide Document: Embedding Binary Data