electron-rare/esp-box
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'docs/update_readme' into 'master'

Browse Source 
docs: update main readme

See merge request ae_group/esp-box!11
This commit is contained in:
Li Bo
2021-11-19 15:14:42 +00:00
parent 5639842c11 67ef6454cb
commit 6779223694
23 changed files with 481 additions and 152 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+36 -73
View File
@@ -1,89 +1,52 @@
* [中文版](./README_cn.md)
* [中文版本](README.md)
# ESP-BOX Developer Guide
# ESP-BOX AIoT Development Framework
ESP-BOX is a new generation AIoT application development kits released by Espressif Systems, powered by Espressif ESP32-S3 SoC, supporting AI acceleration and Wi-Fi + Bluetooth 5 (LE) wireless functions. ESP-BOX has a built-in 2.4-inch LCD capacitive touch screen, dual microphones, a speaker, and dual Pmod™ compatible expansion interfaces. With the out-of-the-box voice wake-up and voice recognition functions, as well as Espressif's open-source development framework and examples, developers can create various interesting AIoT applications better and faster.
**Important Note: We recommend [updating the ESP32-S3-BOX firmware](./docs/firmeware_update.md) when you first receive the product to have the latest features and bugfix.**
## Hardware Overview
The ESP-BOX is a new generation AIoT development platform released by Espressif Systems. The ESP32-S3-BOX is an AIoT development board that is based on Espressif’s ESP32-S3 Wi-Fi + Bluetooth 5 (LE) SoC. The ESP32-S3-BOX provides a platform for developing the control of home appliances using Voice Assistance + touch screen controller, sensor, infrared controller, and intelligent Wi-Fi gateway. The ESP32-S3-BOX comes with pre-built firmware that supports offline voice interaction, with the SDKs and examples provided by Espressif, you will be able to develop a wide variety of AIoT applications based on the ESP32-S3-BOX such as online and offline voice assistants, voice-enabled devices, HMI touch-screen devices, control panels, multi-protocol gateways easily.
![esp_box_hardware](./docs/_static/esp32_s3_box_hardware.svg)
* [ESP-BOX Mainboard Schematic](./docs/hardware/schematic/SCH_ESP32-S3-BOX_V2.5.pdf)
* [ESP-BOX Mainboard PCB](./docs/hardware/pcb/PCB_ESP32-S3-BOX_V2.4.pdf)
**ESP-BOX features:**
* Supports far-field voice interaction with 2 mics
* Offline voice wake-up with high wake-up rate
* Offline speech commands recognition in Chinese and English languages with high recognition rate
* Reconfigurable voice commands in Chinese and English languages
* Flexible and reusable GUI framework
* End-to-end IoT development framework ESP-RainMaker
* Pmod™ compatible headers supports expand peripheral modules
### Specifications:
**ESP-BOX Open-Source Content Include:**
![specs](./docs/_table/specs.png)
* [Schematic and PCB Source File](./hardware)
* [Shell 3D Print Source File](./hardware/esp32_s3_box_shell_step)
* [Firmware Source Code](./examples/factory_demo)
* [Examples Source Code](./examples)
### Ports:
> Based on [Apache 2.0](https://github.com/espressif/esp-box/blob/master/LICENSE) open source license, you can modify, develop, or distribute the contents freely.
![ports](./docs/_table/ports.png)
## User Guide
* [Digilent Pmod™ Interface Specification](https://digilent.com/reference/_media/reference/pmod/pmod-interface-specification-1_3_1.pdf)
* For getting start of ESP32-S3-BOX, [Please to the Getting Started](./docs/getting_started.md)
* For voice control and customization，[Please refer to the Voice Assistance Control and Customization](./docs/getting_started.md#voice-assistance-control-and-customization)
* For control the device by graphical user interface, [Please refer to the GUI Guide](./docs/getting_started.md#esp32-s3-box-graphical-user-interface)
* For more detils of ESP32-S3-BOX hardware，[Please refer to ESP-BOX hardware overview](./docs/hardware_overview.md)
* For latest version firmware，[Please refer to the ESP32-S3-BOX firmware update](./docs/firmeware_update.md)
## Technical Architecture
## Developer Guide
The ESP-BOX development framework includes three main layers: system layer, framework layer, and application layer:
* **Step 1**. ESP-IDF (Espressif IoT Development Framework) is required to build the application. If this is your first time using the ESP-IDF, please refer to [ESP-IDF development guide](https://docs.espressif.com/projects/esp-idf/en/release-v4.4/esp32s3/index.html) to understand the open-source drive and components provided by Espressif.
* **Step 2**. Next, setting up development environment. Please refer to [ESP-IDF(release/v4.4) Installation Step by Step](https://docs.espressif.com/projects/esp-idf/en/release-v4.4/esp32s3/get-started/index.html#installation-step-by-step).
* **Step 3**. Additionally, apply the patch files on current ESP-IDF, please refer [Patch for ESP-IDF](./idf_patch) for details.
* **Step 4**. Take a small step，you can [build and flash a simple example](/examples/image_display).
* **Step 5**. Moving further，please read [ESP-BOX technical architecture](./docs/technical_architecture.md) to understand the technical details.
* **Step 6**. Going deeper，please read [ESP Speech Recognition Application Guide](https://github.com/espressif/esp-sr), [ESP RainMaker Programming Guide](https://docs.espressif.com/projects/esp-rainmaker/en/latest/), [LVGL GUI Quick overview](https://docs.lvgl.io/latest/en/html/get-started/quick-overview.html).
* **Step 7**. For extended function, please read [ESP-BOX Pmod™ compatible headers](./docs/hardware_overview.md) and [ESP-IDF API reference](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/index.html)，to develop drivers for more extended devices.
![](./docs/_static/esp-box-tech-architecture.svg)
## Contact Us
### System Layer
ESP-IDF is the open-source development framework for Espressif SoCs including ESP32、ESP32-S and ESP32-C series. It integrates a large number of system-level basic components, a set of development tools for code compilation and debugging, and detailed development guidance documents:
1. System-level basic components, mainly including multi-tasking, multi-core support based on FreeRTOS, and memory heap allocator with external expansion of PSRAM; a uniform operation interface support for LL, HAL, Driver, and VFS layers of multiple ESP chips; multiple standard network protocol stacks like TCP/IP, HTTP, MQTT, WebSocket, etc.
2. Set of development tools, mainly including GCC cross-toolchain for code building, JTAG debugging tool based on OpenOCD, real-time tracking based on Segger SystemView, Flash and eFuse programmers, etc.
3. Development guidance documents, API references, guidelines, and notes of Espressif’s various chip platforms and software versions. Developers can view these documents [Online](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/) or [Download](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/esp-idf-en-v4.4-dev-3540-g4e03a9c-esp32.pdf).
### Framework Layer
In addition to system-level support, ESP-BOX also integrates several excellent solution frameworks for typical problems in different scenarios. Including the speech recognition framework ESP-SR, the human-computer interaction framework ESP-HMI, and the end-to-end IoT development framework ESP-RainMaker. The functions and customization methods of these solution frameworks will be briefly introduced as follows:
**ESP-SR**
ESP-SR is a speech recognition framework developed by Espressif for the field of intelligent speech. ESP-SR can provide high-performance and high-reliability speech recognition functions for developers by using ESP32-S3 AI extension instructions. Now, this framework has three modules:
1. Espressif's acoustic algorithm ([Audio Front-End, AFE](https://github.com/espressif/esp-sr/tree/master/acoustic_algorithm)) , which is specially designed to improve speech recognition performance in far-field or noisy environments to enable developers to obtain high-quality and stable audio data. It mainly includes AEC (Acoustic Echo Cancellation) , BSS (Blind Source Separation), NS (Noise Suppression). ESP-SR encapsulates the above algorithms into simple APIs. Without undersatanding details of these algorithms, developers only need to arrange and combine the algorithms that need to be used for specific application scenarios, and input the voice data to be processed according to the API format requirements and can get the results.
2. Espressif wake word engine ([WakeNet](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/README.md)), which is used to monitor specific language fragments in real-time in a continuous audio stream, and wake up the smart device from the standby state to the voice interaction state. WakeNet uses MFC (Mel-Frequency Cepstrum) to obtain the features of the input audio clip , and classifies the characteristic signals by the neural network algorithm optimized for ESP32-S3. It can trigger the wake-up signal when monitor the command word, and the [Effective recognition rate](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/README.md#performance-test) is not less than 80% in noisy environment. ESP-BOX provides different offline wake words like “Xiao Ai Tong Xue” “Hi, Lexin” etc. and also allows [customized wake word](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/ESP_Wake_Words_Customization.md).
3. Espressif wake-up word engine ([WakeNet](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/README.md)), which is used to monitor specific language fragments in real-time in a continuous audio stream, and wake up the smart device from the standby state to the voice interaction state. WakeNet uses MFC (Mel-frequency Cepstrum) to obtain the features of the input audio clip, then classifies the characteristic signals by the neural network algorithm optimized for ESP32-S3. That can trigger the wake-up signal when analyzed the command word, and the [Effective recognition rate](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/README.md#performance-test) is more than 80% in a noisy environment. ESP-BOX provides different offline wake words like “Hi, ESP” and “Hi, Lexin”. And also allows [customized wake word](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/ESP_Wake_Words_Customization.md).
**ESP-HMI**
ESP-HMI is a human-computer interaction solution designed by Espressif based on the open-source GUI framework LVGL. With the powerful computing and peripheral resources in ESP32-S3, it can achieve excellent data visualization, touch control, and other functions:
1. Screen driver library: compatible with a variety of mainstream 8080, SPI, I2C interface color screens, monochrome screens, touch screens, and provide a uniform display driver interface. Developers can directly use the adapted screen controller or add their screens according to the display driver interface. And the UI interface can be deployed to a variety of screens.
2. Open source UI templates: Espressif has designed multiple open source UI templates suitable for IoT control panels, multimedia players, voice assistants, and other different IoT scenarios, which can help developers carry out more professional secondary development.
3. Hardware design reference: Espressif provides hardware reference designs, circuit schematics, and development guides related to screen applications to help developers quickly complete hardware designs.
**ESP-RainMaker**
[ESP-RainMaker](https://rainmaker.espressif.com/docs/get-started.html) is a complete system for IoT development, and it can As an end-to-end platform, it can provide developers with one-stop IoT product functions and examples such as device distribution, cloud connection, and OTA, it contains the following functions:
1. Device-side SDK, developers can use Espressif chips to define their IoT products by designing device parameters and attributes, and complete device-side firmware development quickly.
2. Secure cloud middleware, developers do not need to manage device certificates and cloud infrastructure. They can directly use Espressif chips to quickly establish a secure communication link with the cloud.
3. iOS / Android APP, which integrates the necessary functions such as device discovery, Wi-Fi provisioning, user login, device association and control, and developers do not need to carry out additional code development work.
**AI Instructions and Libraries**
With the help of ESP32-S3 Xtensa® 32bit LX7 dual-core CPU and AI / DSP extended instruction set, ESP32-S3 can efficiently complete vector operations, complex number operations, FFT and other operations, which can accelerate neural network calculations and digital signal processing calculation capabilities. AI developers can implement high-performance image recognition, voice wake-up and recognition with Espressif's open source software library based on the AI / DSP extended instruction set.
**Device Drivers**
TBD:
### Application Layer
TBD:
## Develop Guide
1. [Setting up development environment]().
2. Try out some [examples](./examples).
3. Read the [Build System](https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/api-guides/build-system.html) to see the overview of building process.
4. Read the [Application Startup Flow](https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/api-guides/startup.html) to get a better understanding of the program.
5. Read the [Setup Command word]() to redesign your own command words.
6. Read the [LVGL Quick overview](https://docs.lvgl.io/latest/en/html/get-started/quick-overview.html) to learn the most importance things to add your favorite GUI.
7. Read the [ESP-IDF API Reference](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/index.html) to develop drivers for Pmod interface devices.
8. If you have questions please add [GitHub Issues](https://docs.github.com/en/issues) or goto esp32.com forum to get help.
9. [Pull Requests](https://docs.github.com/en/github/collaborating-with-pull-requests) is welcome if you're interested in contributing to ESP-Box.
* Should you have any technical issues, please submit a issue ticket at [GitHub Issues](https://github.com/espressif/esp-box/issues) or forum esp32.com for help.
* If you are interested with ESP-BOX project，welcome to the [ESP-BOX forum](https://esp32.com/viewforum.php?f=44) to share with us your idea.
* We welcome any open-source software or hardware contribution. Please commit to the [Pull Requests](https://github.com/espressif/esp-box/pulls) to contrinute your code.
README_cn.md
+37 -70
View File
@@ -1,86 +1,53 @@
* [English Version](./README.md)
# ESP-BOX 开发者指南
# ESP-BOX AIoT 开发框架
ESP-Box 是乐鑫信息科技发布的新一代 AIoT 应用开发板，搭载乐鑫 ESP32-S3 SoC，支持 AI 加速指令和 Wi-Fi + Bluetooth 5 (LE) 无线功能。ESP-Box 内置 2.4 英寸 LCD 电容式触摸显示屏，双麦克风，扬声器，和两个 Pmod 兼容的多功能扩展接口。配合开箱即用的语音唤醒和识别功能，以及开源开发框架和示例程序，开发者基于 ESP-Box 可以更快更好的构建各种有趣的 AIoT 智能应用。
**重要提醒：**
**我们建议您在第一次收到产品时[更新 ESP32-S3-BOX 的固件](./docs/firmeware_update_cn.md)以获得最新功能和 bug 的修复。**
## 硬件总览
ESP-BOX 是乐鑫信息科技发布的新一代 AIoT 应用开发平台。ESP32-S3-BOX 是对应的 AIoT 应用开发板，搭载支持 AI 加速的 ESP32-S3 Wi-Fi + Bluetooth 5 (LE) SoC。ESP32-S3-BOX 为用户提供了一个基于语音助手 + 触摸屏控制、传感器、红外控制器和智能 Wi-Fi 网关等功能，开发和控制智能家居设备的平台。ESP32-S3-BOX 出厂支持离线语音交互功能，通过乐鑫丰富的 SDK 和解决方案，能够轻松构建在线和离线语音助手、智能语音设备、HMI 触摸屏设备、控制面板、多协议网关等多样的应用。
![esp_box_hardware](./docs/_static/esp32_s3_box_hardware.svg)
* [ESP-BOX Mainboard Schematic](./docs/hardware/schematic/SCH_ESP32-S3-BOX_V2.5.pdf)
* [ESP-BOX Mainboard PCB](./docs/hardware/pcb/PCB_ESP32-S3-BOX_V2.4.pdf)
**ESP-BOX 支持以下特性:**
* 双麦克风支持远场语音交互
* 高唤醒率的离线语音唤醒
* 高识别率的离线中英文命令词识别
* 可动态配置的中英文命令词
* 灵活可复用的 GUI 框架
* 端到端一站式接入云平台
* Pmod™ 兼容接口支持多种外设扩展
### 规格参数:
**ESP-BOX 开源内容包括:**
![specs](./docs/_table/specs.png)
* [原理图与 PCB 源文件](./hardware)
* [外壳 3D 打印源文件](./hardware/esp32_s3_box_shell_step)
* [出厂固件源代码](./examples/factory_demo)
* [示例程序源代码](./examples)
### 接口:
> 基于 [Apache 2.0](https://github.com/espressif/esp-box/blob/master/LICENSE) 开源协议，你可以免费且自由的修改、创作和分享。
![ports](./docs/_table/ports.png)
## 使用指引
* [Digilent Pmod™ Interface Specification](https://digilent.com/reference/_media/reference/pmod/pmod-interface-specification-1_3_1.pdf)
## 技术架构
ESP-BOX 开发框架包括系统层、框架层、应用层三个主要层级：
![](./docs/_static/esp-box-tech-architecture.svg)
### 系统层 System Layer
ESP-IDF 是乐鑫官方开源操作系统框架，同时兼容 ESP32、ESP32-S 和 ESP32-C 系列 SoC，完成一次开发即可在多芯片平台部署。它集成了大量的系统级基础组件，用于代码编译和调试的开发工具集，以及详尽的开发指导文档，具体包括：
1. 系统级基础组件，主要包括基于 FreeRTOS 的多任务、多核支持，可外扩 PSRAM 的内存堆分配器；多款 ESP 芯片的 LL、HAL、Driver、VFS 层支持，将不同芯片功能抽象为统一操作接口；多个标准网络协议栈 TCP/IP、HTTP、MQTT、WebSocket 等等；
2. 开发工具集，主要包含用于代码编译的 GCC 交叉工具链、基于 OpenOCD 的 JTAG 调试工具、基于 Segger SystemView 的实时跟踪、Flash 和 eFuse 编程器等；
3. 开发指导文档，详尽的说明了乐鑫各个芯片平台、各个软件版本下的 API 参考，使用指引和注意事项等，开发者可[在线浏览](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/)也可[下载到本地](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/esp-idf-en-v4.4-dev-3540-g4e03a9c-esp32.pdf)查看。
### 框架层 Framework Layer
除了系统层支持外，ESP-BOX 还集成了多个出色的解决方案框架，分别处理不同功能场景下的典型问题。包括用于处理语音唤醒词和命令词的语音识别框架 ESP-SR，处理显示和触摸交互功能的人机交互框架 ESP-HMI，和处理物联网功能的端到端物联网开发框架 ESP-RainMaker，接下来将简要介绍这些解决方案框架的功能和定制化方法：
**ESP-SR**
ESP-SR 是乐鑫面向智能语音领域开发的语音识别框架，该框架可利用 ESP32-S3 AI 扩展指令进行运算加速，为开发者提供开箱即用的高性能、高可靠性的语音识别功能，它包括：
1. 声学前端（[Audio Front-End, AFE](https://github.com/espressif/esp-sr/tree/master/acoustic_algorithm)）算法集，用于提升智能语音设备在远场噪声环境中的交互能力，使开发者获得高质量且稳定的音频数据。它主要包括可以有效滤除扬声器回声的回声消除算法（Acoustic Echo Cancellation，AEC）, 用于强化麦克风阵列方向性的盲源分离算法（Blind Source Separation，BSS），和用于处理环境噪声的噪声抑制算法（Noise Suppression，NS）。ESP-SR 将以上算法封装成简单的 API，开发者无需关心这些算法的具体细节，只需要针对具体的应用场景，对需要使用的算法排列组合，并按照 API 的格式要求输入待处理的语音数据，即可得到声学前端算法的处理结果。
2. 唤醒词识别模型（[WakeNet](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/README.md)），用于在连续的音频流中实时监测出特定的语言片段，将智能设备从待机状态唤醒至语音交互状态。WakeNet 通过计算音频片段的梅尔倒频谱（Mel-Frequency Cepstrum, MFC）作为输入特征，使用针对 ESP32-S3 优化的神经网络算法对特征信号进行分类，在监测到命令词时触发唤醒信号，可在噪声环境下获得不小于 80% 的[有效识别率](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/README.md#performance-test)。ESP-BOX 提供了开箱即用的离线唤醒词 “Hi, ESP” 和 “Hi, 乐鑫”，开发者无需额外的开发工作即可使用，同时乐鑫也支持为开发者[定制独特的唤醒词](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/ESP_Wake_Words_Customization.md)。
3. 命令词识别模型（[MultiNet](https://github.com/espressif/esp-sr/blob/master/speech_command_recognition/README.md)），用于在设备唤醒后，识别用户特定的中英文语音命令，例如 “Turn on the air conditioner”，“关闭电灯” 等。MultiNet 通过基于 CRNN 和 CTC 的卷积循环神经网络，对输入音频片段进行多命令词识别，可同时支持最多 100 个离线命令词。 开发者不需要了解识别模型细节，也无需重新进行模型训练，通过简单配置[中文拼音字符串](https://github.com/espressif/esp-sr/blob/master/speech_command_recognition/README.md#chinese-speech-command-recognition)或[英文音标字符串](https://github.com/espressif/esp-sr/blob/master/speech_command_recognition/README.md#english-speech-command-recognition)以及对应 ID，即可添加或修改命令词，模型在识别到语音命令词后将触发包含 ID 信息的事件。
**ESP-HMI**
ESP-HMI 是乐鑫基于开源 GUI 框架 LVGL 设计的人机交互方案，借助 ESP32-S3 强大的计算能力和外设资源，可以实现出色的数据可视化、触摸控制等功能，它包括：
1. 屏幕驱动库：兼容多款主流 8080、SPI、I2C 接口彩色屏幕、单色屏、触摸屏，提供统一的显示驱动接口。开发者可以直接使用已适配的屏幕控制器，也可根据显示驱动接口添加自己的屏幕，UI 界面开发完成可无缝部署到多种屏幕。
2. 开源 UI 模板：乐鑫设计了适用于物联网控制面板、多媒体播放器、语音助手等不同物联网场景的多套开源 UI 模板，帮助开发者进行更专业的二次开发。
3. 硬件参考设计：提供屏幕应用相关的硬件参考设计、电路原理图和开发指南，帮助开发者快速完成硬件设计。
**ESP-RainMaker**
[ESP-RainMaker](https://rainmaker.espressif.com/docs/get-started.html) 是一个完整的物联网开发框架，作为一个端到端平台，可以为开发者提供设备配网、连云、OTA 等一站式的物联网产品功能和示例。它包含：
1. 设备端 SDK，开发者通过设计设备参数和属性，即可使用乐鑫芯片定义自己的物联网产品，快速完成设备端固件开发;
2. 透明的云中间件，开发者无需管理设备证书和云端基础设施，直接使用乐鑫芯片即可快速建立与云端的建立安全的通信链接；
3. iOS / Android 平台 APP，集成了设备发现，Wi-Fi 配网，用户登录，设备关联和控制等必要功能，开发者无需进行额外的代码开发工作。
**AI 指令集和库**
借助 ESP32-S3 Xtensa® 32­bit LX7 双核 CPU 和 AI / DSP 扩展指令集，ESP32-S3 可高效完成向量运算、复数运算、FFT 等运算，加速神经网络计算和数字信号处理计算能力。AI 开发者们可以基于乐鑫开源的基于 AI / DSP 扩展指令集的软件库，实现高性能的图像识别、语音唤醒与识别等应用。
**设备驱动 Device Drivers**
TBD:
### 应用层 Application Layer
TBD:
* 首次拿到 ESP-BOX，[您可以查看首次使用操作指引](./docs/getting_started_cn.md#给-esp32-s3-box-供电)
* 想要设置独特的语音命令词，[您可以查看修改自定义命令词](./docs/getting_started_cn.md#语音控制命令词自定义)
* 想要通过界面控制设备，[请查看图形界面操作指引](./docs/getting_started_cn.md#esp32-s3-box-图形界面)
* 想要了解 ESP-BOX 硬件和接口细节，[请查看ESP-BOX 硬件总览](./docs/hardware_overview_cn.md)
* 想要使用最新版本固件，[请查看固件更新说明](./docs/firmeware_update_cn.md)
## 开发指引
1. [搭建开发环境]()；
2. 尝试编译几个[示例程序](./examples)；
3. 查阅 [编译系统](https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/api-guides/build-system.html) 可以进一步理解程序的编译链接过程；
4. 查阅 [应用程序启动流程](https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/api-guides/startup.html) 可以更好的理解 ESP 固件程序的运行流程；
5. 查阅 [配置命令词]() 定义自己的专属命令词；
6. 查阅 [LVGL 快速总览](https://docs.lvgl.io/latest/en/html/get-started/quick-overview.html) 学习如何添加一个喜爱的 GUI；
7. 查阅 [ESP-IDF API 参考手册](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/index.html) 为 Pmod 外接设备开发驱动；
8. 如果你有问题，可添加 [GitHub Issues](https://docs.github.com/en/issues) 或者到 esp32.com 论坛寻求帮助；
9. 如果你对 ESP-BOX 项目感兴趣，欢迎贡献代码 [Pull Requests](https://docs.github.com/en/github/collaborating-with-pull-requests)。
* **Step 1**. 如果您首次接触 ESP-IDF 开发，建议先浏览 [ESP-IDF 编程指南](https://docs.espressif.com/projects/esp-idf/en/release-v4.4/esp32s3/index.html)，了解乐鑫已经提供的开源驱动和组件；
* **Step 2**. 接下来您可以详细阅读 [ESP-IDF（release/v4.4） 环境搭建指引](https://docs.espressif.com/projects/esp-idf/en/release-v4.4/esp32s3/get-started/index.html#installation-step-by-step), 一步一步完成开发环境搭建；
* **Step 3**. 此外, 请在当前 ESP-IDF 上打上补丁, 详情请参考 [Patch for ESP-IDF](./idf_patch).
* **Step 4**. 小试牛刀，您可以尝试 [构建并烧录一个新的示例程序](/examples/image_display)；
* **Step 5**. 更进一步，您可以阅读 [ESP-BOX 技术架构说明](./docs/technical_architecture_cn.md)，了解更多技术细节；
* **Step 6**. 深入开发，您可以阅读 [ESP 语音识别应用开发指南](https://github.com/espressif/esp-sr), [ESP RainMaker 编程指南](https://docs.espressif.com/projects/esp-rainmaker/en/latest/), [LVGL UI 界面开发快速总览](https://docs.lvgl.io/latest/en/html/get-started/quick-overview.html)；
* **Step 7**. 扩展功能，您可以阅读 [ESP-BOX Pmod™ 兼容接口介绍](./docs/hardware_overview_cn.md) 和 [ESP-IDF API 参考手册](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/index.html)，为更多扩展设备开发驱动；
## 联系我们
* 如果有任何开发和使用问题，您可添加 [GitHub Issues](https://github.com/espressif/esp-box/issues) 或到 esp32.com 论坛寻求帮助；
* 如果你对 ESP-BOX 项目感兴趣，欢迎到论坛 [ESP-BOX 版块](https://esp32.com/viewforum.php?f=43)交流想法；
* 我们欢迎任何软件或硬件相关的开源贡献，您可直接提交 [Pull Requests](https://github.com/espressif/esp-box/pulls) 贡献代码。
docs/_static/bin_download.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 72 KiB

docs/_static/device_manager_usb_serial_jtag.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 60 KiB

docs/_static/device_manager_usb_serial_jtag_cn.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 21 KiB

docs/_static/dl_tool_windows.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.4 KiB

docs/_static/dl_tool_windows_2.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 60 KiB

docs/_static/dl_tool_windows_3.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

docs/_static/esp-box-tech-architecture.svg
Vendored
+1 -1
View File
File diff suppressed because one or more lines are too long
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 30 KiB

After

Width:  |  Height:  |  Size: 33 KiB

docs/_static/esp32_s3_box_hardware.svg
Vendored
+1 -1
View File
File diff suppressed because one or more lines are too long
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 63 KiB

After

Width:  |  Height:  |  Size: 32 KiB

docs/_static/linux_download.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 44 KiB

docs/_static/linux_install_esptool.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 54 KiB

docs/_static/macos_download.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 371 KiB

docs/_static/macos_install_esptool.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 720 KiB

docs/_static/page_use_intr.png
Vendored
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 8.9 KiB

docs/firmeware_update.md
+120
View File
@@ -0,0 +1,120 @@
* [中文版本](./firmeware_update_cn.md)
## Update ESP32-S3-BOX Firmware
ESP32-S3-BOX supports updating firmware through USB, you just need:
1. USB Type-C cable
2. [Latest firmware bin file](https://github.com/espressif/esp-box/releases)
### Download Latest Firmware
For latest firmware, please refer to: https://github.com/espressif/esp-box/releases
1. Choose latest release version
2. Click `Assets` to show more files
3. Choose a `xxxxxx.bin` to download
<div align="center">
<img src="_static/bin_download.png">
</div>
### Firmware update for Windows
We recommend using `Windows 10` and above system. Under `Windows 10` system, the driver of `USB-Serial-Jtag` will be downloaded automatically. if you use the `Windows 7`, please download and install [USB-Serial-JTAG drive](https://dl.espressif.com/dl/idf-driver/idf-driver-esp32-usb-jtag-2021-07-15.zip) manully.
1. [Download latest firmware](https://github.com/espressif/esp-box/releases)
2. Connect ESP32-S3-BOX to the computer through USB Type-C cable.
<div align="center">
<img src="_static/plug_power.png">
</div>
3. Please make sure the computer is connected to the Internet firstly. When the driver is installed, you can find the devices in `Device Manager`. There will be two more new devices `COMX` (`COM2` for example) and `USB JTAG/serial debug unit`, the former is used to download firmware or output program logs, the latter is used for JTAG debugging.
<div align="center">
<img src="_static/device_manager_usb_serial_jtag.png">
</div>
4. Download [Windows download tool](https://www.espressif.com/sites/default/files/tools/flash_download_tool_3.9.2_0.zip) and unzip it to any folder, then run the executable file `flash_download_tool_x.x.x.exe`
5. Please choose `chipType`: `ESP32S3`, `workMode`: `develop`, `loadMode`: `usb`, then click `OK` to enter the download ui:
<div align="center">
<img src="_static/dl_tool_windows.png">
</div>
6. Configure the download tool follow the instruction below:
1. Choose the path of firmwire `xxxx.bin` , and set the address to `0x0`
2. Select the COM port, like `COM2` for this PC.
3. Click `START` to start download.
<div align="center">
<img src="_static/dl_tool_windows_2.png">
</div>
7. After downloading, `FINISH` will shown in the tool, then **please reboot to run the new firmware!**
<div align="center">
<img src="_static/dl_tool_windows_3.png">
</div>
### Firmware update for Linux
1. [Download latest firmware](https://github.com/espressif/esp-box/releases)
2. Connect ESP32-S3-BOX to the computer through USB Type-C cable，there is no need to install the driver of `USB-Serial-Jtag` under Linux system.
3. Install `esptool`, input following commands in `Terminal` (`pip` can be specified as `pip3`) :
```
pip install esptool
```
<div align="center">
<img src="_static/linux_install_esptool.png">
</div>
4. Download firmware using following instruction (`pip` can be specified as `pip3`)：
```
python -m esptool --chip esp32s3 write_flash 0x0 download_path/test_bin.bin
```
1. `0x0` is the flash address
2. `download_path/test_bin.bin` is the name and path the downloaded firmware.
5. After updating, download tool will prompt `Hash of data verified`, then **please reboot to run the new firmware!**
<div align="center">
<img src="_static/linux_download.png">
</div>
### Firmware update for macOS
1. [Download latest firmware](https://github.com/espressif/esp-box/releases)
2. Connect ESP32-S3-BOX to the computer through USB Type-C cable，there is no need to install the driver of `USB-Serial-Jtag` under macOS.
3. Install `esptool`, input following codes in `Terminal` (`pip3` can be specified as `pip`) :
```
pip3 install esptool
```
<div align="center">
<img src="_static/macos_install_esptool.png">
</div>
4. Download firmware using following commands (`pip3` can be specified as `pip`):
```
python3 -m esptool --chip esp32s3 write_flash 0x0 download_path/test_bin.bin
```
1. `0x0` is the flash address
2. `download_path/test_bin.bin` is the name and path for the downloaded firmware.
5. After updating, download tool will prompt `Hash of data verified`, then **please reboot to run the new firmware!**
<div align="center">
<img src="_static/macos_download.png">
</div>
docs/firmeware_update_cn.md
+128
View File
@@ -0,0 +1,128 @@
* [English Version](./firmeware_update.md)
## 更新 ESP32-S3-BOX 固件
ESP32-S3-BOX 支持使用 USB 接口更新固件，你仅需要准备：
1. 一根 USB Type-C 数据线
2. 最新版本固件（bin 文件）
### 下载最新版本固件
请下载最新版本固件，固件下载链接为：https://github.com/espressif/esp-box/releases
1. 请先选择一个固件（建议使用最新 release）
2. 点击 `Assets` 展开下载链接
3. 选择一个 `xxxxxx.bin` 进行下载
<div align="center">
<img src="_static/bin_download.png">
</div>
### Windows 系统更新固件说明
我们推荐使用 `Windows 10` 及以上版本，在该系统下 `USB-Serial-Jtag` 的驱动将联网自动下载。如果使用 `Windows 7` 系统，请手动下载 [USB-Serial-JTAG 驱动](https://dl.espressif.com/dl/idf-driver/idf-driver-esp32-usb-jtag-2021-07-15.zip) 并安装。
1. [下载最新版本固件](https://github.com/espressif/esp-box/releases)
2. 使用 USB Type-C 数据线将 ESP32-S3-BOX 接入电脑
<div align="center">
<img src="_static/plug_power.png">
</div>
3. 初次使用，请确保电脑已联网，驱动正常自动安装后，我们能在设备管理器看到以下设备。这里将多出两个新的设备 `COMX`(此电脑为 `COM2` ) 和 `USB JTAG/serial debug unit`，前者用于下载固件和输出程序日志，后者用于 JTAG 调试
<div align="center">
<img src="_static/device_manager_usb_serial_jtag_cn.png">
</div>
4. 下载 [Windows download tool](https://www.espressif.com/sites/default/files/tools/flash_download_tool_3.9.2_0.zip)，并解压到任意文件夹，然后请双击打开下载工具可执行文件 `flash_download_tool_x.x.x.exe`
5. 请选择 `chipType`: `ESP32S3`, `workMode`: `develop`, `loadMode`: `usb`，之后点击 `OK` 进入下载工具界面：
<div align="center">
<img src="_static/dl_tool_windows.png">
</div>
6. 请按照下图指示配置下载工具:
1. 首先选择 `xxxx.bin` 路径，将地址设置为 `0x0`
2. 选择下载端口，此电脑为 `COM2`
3. 点击 `START` 开始固件下载
<div align="center">
<img src="_static/dl_tool_windows_2.png">
</div>
7. 下载完成后，工具将提示 `FINISH`，之后**重新上电，即可进入新程序！**
<div align="center">
<img src="_static/dl_tool_windows_3.png">
</div>
### Linux 系统更新固件说明
1. [下载最新版本固件](https://github.com/espressif/esp-box/releases)
2. 使用 USB Type-C 数据线将 ESP32-S3-BOX 接入电脑，`USB-Serial-Jtag` 在 Linux 系统下无需安装驱动
<div align="center">
<img src="_static/plug_power.png">
</div>
3. 安装下载工具 `esptool`，请打开 `终端` ，并输入以下指令（`pip` 也可指定为 `pip3`）：
```
pip install esptool
```
<div align="center">
<img src="_static/linux_install_esptool.png">
</div>
4. 请使用以下指令下载固件（`python` 也可指定为 `python3`）：
```
python -m esptool --chip esp32s3 write_flash 0x0 download_path/test_bin.bin
```
1. `0x0` 是即将写入的 flash 地址
2. `download_path/test_bin.bin` 为固件下载的路径和名称
5. 下载完成后，工具将提示 `Hash of data verified`，之后**重新上电，即可进入新程序！**
<div align="center">
<img src="_static/linux_download.png">
</div>
### macOS 系统更新固件说明
1. [下载最新版本固件](https://github.com/espressif/esp-box/releases)
2. 使用 USB Type-C 数据线将 ESP32-S3-BOX 接入电脑，`USB-Serial-Jtag` 在 macOS 系统下无需安装驱动
<div align="center">
<img src="_static/plug_power.png">
</div>
3. 安装下载工具 `esptool`，请打开 `终端` ，并输入以下指令（`pip3` 也可指定为 `pip`）：
```
pip3 install esptool
```
<div align="center">
<img src="_static/macos_install_esptool.png">
</div>
4. 请使用以下指令下载固件（`python3` 也可指定为 `python`）：
```
python3 -m esptool --chip esp32s3 write_flash 0x0 download_path/test_bin.bin
```
1. `0x0` 是即将写入的 flash 地址
2. `download_path/test_bin.bin` 为固件下载的路径和名称
5. 下载完成后，工具将提示 `Hash of data verified`，之后**重新上电，即可进入新程序！**
<div align="center">
<img src="_static/macos_download.png">
</div>
docs/getting_started.md
+5 -5
View File
@@ -2,7 +2,7 @@
# Getting Started
The ESP32-S3-BOX provides a platform for developing the control of home appliances using Voice Assistance + touch screen controller, sensor, infrared controller, and intelligent Wi-Fi gateway. The ESP32-S3-BOX comes with pre-built firmware that allows you to experience voice wake-up and offline speech commands recognition in Chinese and English languages, with the SDKs and examples provided by Espressif, you will be able to develop a wide variety of AIoT applications based on the ESP32-S3-BOX with great ease, such as online and offline voice assistants, voice-enabled devices, HMI touch-screen devices, control panels, multi-protocol gateways, and so on.
The ESP32-S3-BOX provides a platform for developing the control of home appliances using Voice Assistance + touch screen controller, sensor, infrared controller, and intelligent Wi-Fi gateway. The ESP32-S3-BOX comes with pre-built firmware that allows you to experience voice wake-up and offline speech commands recognition in Chinese and English languages, with the SDKs and examples provided by Espressif, you will be able to develop a wide variety of AIoT applications based on the ESP32-S3-BOX such as online and offline voice assistants, voice-enabled devices, HMI touch-screen devices, control panels, multi-protocol gateways easily.
The ESP32-S3-BOX SDK also includes a reconfigurable AI voice interaction that allows you to customize commands to control your smart home appliances. The content of this material is to give you a brief concept of how the ESP32-S3-BOX works, a basic idea for you to know where to begin. Once you go through the guide below, you may start developing the application yourself. So, let’s begin the journey!
@@ -59,7 +59,7 @@ The ESP32-S3-BOX SDK also includes a reconfigurable AI voice interaction that al
<img src="./_static/page_home.png">
</div>
3. Now, say the wake word **“Hi E.S.P.” (pronounce in alphabet)** to activate your ESP32-S3-BOX. If the board wakes up, its screen below will display the wake word you just spoke. If the wake word is not displayed, try the wake word more times. The animation below gives you a hint that your device is listening.
3. Now, say the wake word **“Hi E. S. P.” (pronounce in alphabet)** to activate your ESP32-S3-BOX. If the board wakes up, its screen below will display the wake word you just spoke. If the wake word is not displayed, try the wake word more times. The animation below gives you a hint that your device is listening.
<div align="center">
<img src="./_static/page_hi.png">
@@ -159,16 +159,16 @@ Scan another QR code again, and it will redirect you to the web control interfac
<img src="_static/web_config_light_cmd_cg.JPG">
</div>
5. Now, you can try out your new command! First, say **“Hi ESP”** to wake-up your device. Then say **“Good morning”** to turn on your light within 6 seconds. The new command phrase will display on the screen and the LED module will be turned on as below.。
5. Now, you can try out your new command! First, say **“Hi E. S. P.”** to wake-up your device. Then say **“Good morning”** to turn on your light within 6 seconds. The new command phrase will display on the screen and the LED module will be turned on as below.。
<div align="center">
<img src="_static/page_good_morning.png">
</div>
>**How to add appropriate command phrases:**
Now AI model supports users to custom command phrases. In order to obtain the best recognition experience, you need to pay attention to the following matters when defining command phrase:
Now AI model supports users to customize command phrases. In order to obtain the best recognition experience, you need to pay attention to the following matters when defining command phrase:
>* Length of command word
The number of words in the command phrase must be greater than or equal to 2 and less than or equal to 10.
The number of words in the command phrase must be greater than or equal to 2 and less than or equal to 8.
When defining a series of command phrases, it is best to keep the length of different command phrases similar.
>* Avoid prefixes
Multiple command phrase should not be prefixed with each other. Short words will be not recognized, such as “Turn on” and “Turn on the light", “Turn on" will be not recognized.
docs/getting_started_cn.md
+2 -2
View File
@@ -163,7 +163,7 @@ ESP32-S3-BOX 最令人印象深刻的功能是乐鑫专有的 AI 语音识别系
>**如何添加合适的命令词:**
AI 模型支持自定义命令词，为了获得最优的识别体验，命令词定义时需要注意以下事项：
>* 命令词的长度
命令词中单词数需要大于等于 2 个，小于等于 10 个。
命令词中单词数需要大于等于 2 个，小于等于 8 个。
在定义一系列命令词时，最好可以保持不同命令词的长度相似。
>* 避免前缀
多个命令词，相互之间不要是对方的前缀，短的词会被屏蔽，比如 “打开” 和 “打开灯”， “打开”会被屏蔽。
@@ -194,4 +194,4 @@ ESP32-S3-BOX 最令人印象深刻的功能是乐鑫专有的 AI 语音识别系
## ESP-BOX 论坛
访问乐鑫 BBS 论坛 [esp32.com](https://esp32.com/viewforum.php?f=44) 给我们提问，或者通过 sales@espressif.com 邮箱联系我们，我们将尽快答复您。
访问乐鑫 BBS 论坛 [esp32.com](https://esp32.com/viewforum.php?f=43) 给我们提问，或者通过 sales@espressif.com 邮箱联系我们，我们将尽快答复您。
docs/hardware_overview.md
+23
View File
@@ -0,0 +1,23 @@
## Hardware Overview
![esp_box_hardware](_static/esp32_s3_box_hardware.svg)
### Hardware Source Files
* [ESP-BOX Mainboard Schematic](../hardware/esp32_s3_box_v2.5/schematic)
* [ESP-BOX Mainboard PCB](../hardware/esp32_s3_box_v2.5/pcb)
* [ESP-BOX Mainboard Gerber](../hardware/esp32_s3_box_v2.5/gerber)
* [ESP-BOX Subboard Schematic](../hardware/esp32_s3_box_sub_v2.2/schematic)
* [ESP-BOX Subboard PCB](../hardware/esp32_s3_box_sub_v2.2/pcb)
* [ESP-BOX Subboard Gerber](../hardware/esp32_s3_box_sub_v2.2/gerber)
* [ESP-BOX Shell CAD STEP](../hardware/esp32_s3_box_shell_step)
### Specifications:
![specs](_table/specs.png)
### Ports:
![ports](_table/ports.png)
* [Digilent Pmod™ Interface Specification](https://digilent.com/reference/_media/reference/pmod/pmod-interface-specification-1_3_1.pdf)
docs/hardware_overview_cn.md
+23
View File
@@ -0,0 +1,23 @@
## 硬件总览
![esp_box_hardware](_static/esp32_s3_box_hardware.svg)
### 硬件开源资料
* [ESP-BOX Mainboard Schematic](../hardware/esp32_s3_box_v2.5/schematic)
* [ESP-BOX Mainboard PCB](../hardware/esp32_s3_box_v2.5/pcb)
* [ESP-BOX Mainboard Gerber](../hardware/esp32_s3_box_v2.5/gerber)
* [ESP-BOX Subboard Schematic](../hardware/esp32_s3_box_sub_v2.2/schematic)
* [ESP-BOX Subboard PCB](../hardware/esp32_s3_box_sub_v2.2/pcb)
* [ESP-BOX Subboard Gerber](../hardware/esp32_s3_box_sub_v2.2/gerber)
* [ESP-BOX Shell CAD STEP](../hardware/esp32_s3_box_shell_step)
### 规格参数
![specs](_table/specs.png)
### 接口
![ports](_table/ports.png)
* [Digilent Pmod™ Interface Specification](https://digilent.com/reference/_media/reference/pmod/pmod-interface-specification-1_3_1.pdf)
docs/technical_architecture.md
+54
View File
@@ -0,0 +1,54 @@
* [中文版](./technical_architecture_cn.md)
## Technical Architecture
The ESP-BOX development framework includes three main layers: system layer, framework layer, and application layer:
<div align="center">
<img src="_static/esp-box-tech-architecture.svg">
</div>
### System Layer
ESP-IDF is the open-source development framework for Espressif SoCs including ESP32、ESP32-S and ESP32-C series. It integrates a large number of system-level basic components, a set of development tools for code compilation and debugging, and detailed development guidance documents:
1. System-level basic components, mainly including multi-tasking, multi-core support based on FreeRTOS, and memory heap allocator with external expansion of PSRAM; a uniform operation interface support for LL, HAL, Driver, and VFS layers of multiple ESP chips; multiple standard network protocol stacks like TCP/IP, HTTP, MQTT, WebSocket, etc.
2. Set of development tools, mainly including GCC cross-toolchain for code building, JTAG debugging tool based on OpenOCD, real-time tracking based on Segger SystemView, Flash and eFuse programmers, etc.
3. Development guidance documents, API references, guidelines, and notes of Espressif’s various chip platforms and software versions. Developers can view these documents [Online](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/) or [Download](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/esp-idf-en-v4.4-dev-3540-g4e03a9c-esp32.pdf).
### Framework Layer
In addition to system-level support, ESP-BOX also integrates several excellent solution frameworks for typical problems in different scenarios. Including the speech recognition framework ESP-SR, the human-computer interaction framework ESP-HMI, and the end-to-end IoT development framework ESP-RainMaker. The functions and customization methods of these solution frameworks will be briefly introduced as follows:
**ESP-SR**
ESP-SR is a speech recognition framework developed by Espressif for the field of intelligent speech. ESP-SR can provide high-performance and high-reliability speech recognition functions for developers by using ESP32-S3 AI extension instructions. Now, this framework has three modules:
1. Espressif's acoustic algorithm ([Audio Front-End, AFE](https://github.com/espressif/esp-sr/tree/master/acoustic_algorithm)) , which is specially designed to improve speech recognition performance in far-field or noisy environments to enable developers to obtain high-quality and stable audio data. It mainly includes AEC (Acoustic Echo Cancellation) , BSS (Blind Source Separation), NS (Noise Suppression). ESP-SR encapsulates the above algorithms into simple APIs. Without undersatanding details of these algorithms, developers only need to arrange and combine the algorithms that need to be used for specific application scenarios, and input the voice data to be processed according to the API format requirements and can get the results.
2. Espressif wake word engine ([WakeNet](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/README.md)), which is used to monitor specific language fragments in real-time in a continuous audio stream, and wake up the smart device from the standby state to the voice interaction state. WakeNet uses MFC (Mel-Frequency Cepstrum) to obtain the features of the input audio clip , and classifies the characteristic signals by the neural network algorithm optimized for ESP32-S3. It can trigger the wake-up signal when monitor the command word, and the [Effective recognition rate](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/README.md#performance-test) is not less than 80% in noisy environment. ESP-BOX provides different offline wake words like “Xiao Ai Tong Xue” “Hi, Lexin” etc. and also allows [customized wake word](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/ESP_Wake_Words_Customization.md).
3. Espressif wake-up word engine ([WakeNet](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/README.md)), which is used to monitor specific language fragments in real-time in a continuous audio stream, and wake up the smart device from the standby state to the voice interaction state. WakeNet uses MFC (Mel-frequency Cepstrum) to obtain the features of the input audio clip, then classifies the characteristic signals by the neural network algorithm optimized for ESP32-S3. That can trigger the wake-up signal when analyzed the command word, and the [Effective recognition rate](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/README.md#performance-test) is more than 80% in a noisy environment. ESP-BOX provides different offline wake words like “Hi, ESP” and “Hi, Lexin”. And also allows [customized wake word](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/ESP_Wake_Words_Customization.md).
**ESP-HMI**
ESP-HMI is a human-computer interaction solution designed by Espressif based on the open-source GUI framework LVGL. With the powerful computing and peripheral resources in ESP32-S3, it can achieve excellent data visualization, touch control, and other functions:
1. Screen driver library: compatible with a variety of mainstream 8080, SPI, I2C interface color screens, monochrome screens, touch screens, and provide a uniform display driver interface. Developers can directly use the adapted screen controller or add their screens according to the display driver interface. And the UI interface can be deployed to a variety of screens.
2. Open source UI templates: Espressif has designed multiple open source UI templates suitable for IoT control panels, multimedia players, voice assistants, and other different IoT scenarios, which can help developers carry out more professional secondary development.
3. Hardware design reference: Espressif provides hardware reference designs, circuit schematics, and development guides related to screen applications to help developers quickly complete hardware designs.
**ESP-RainMaker**
[ESP-RainMaker](https://rainmaker.espressif.com/docs/get-started.html) is a complete system for IoT development, and it can As an end-to-end platform, it can provide developers with one-stop IoT product functions and examples such as device distribution, cloud connection, and OTA, it contains the following functions:
1. Device-side SDK, developers can use Espressif chips to define their IoT products by designing device parameters and attributes, and complete device-side firmware development quickly.
2. Secure cloud middleware, developers do not need to manage device certificates and cloud infrastructure. They can directly use Espressif chips to quickly establish a secure communication link with the cloud.
3. iOS / Android APP, which integrates the necessary functions such as device discovery, Wi-Fi provisioning, user login, device association and control, and developers do not need to carry out additional code development work.
**AI Instructions and Libraries**
With the help of ESP32-S3 Xtensa® 32bit LX7 dual-core CPU and AI / DSP extended instruction set, ESP32-S3 can efficiently complete vector operations, complex number operations, FFT and other operations, which can accelerate neural network calculations and digital signal processing calculation capabilities. AI developers can implement high-performance image recognition, voice wake-up and recognition with Espressif's open source software library based on the AI / DSP extended instruction set.
**Device Drivers**
### Application Layer
docs/technical_architecture_cn.md
+51
View File
@@ -0,0 +1,51 @@
* [English Version](./technical_architecture.md)
## 技术架构
ESP-BOX 开发框架包括系统层、框架层、应用层三个主要层级：
<div align="center">
<img src="_static/esp-box-tech-architecture.svg">
</div>
### 系统层 System Layer
ESP-IDF 是乐鑫官方开源操作系统框架，同时兼容 ESP32、ESP32-S 和 ESP32-C 系列 SoC，完成一次开发即可在多芯片平台部署。它集成了大量的系统级基础组件，用于代码编译和调试的开发工具集，以及详尽的开发指导文档，具体包括：
1. 系统级基础组件，主要包括基于 FreeRTOS 的多任务、多核支持，可外扩 PSRAM 的内存堆分配器；多款 ESP 芯片的 LL、HAL、Driver、VFS 层支持，将不同芯片功能抽象为统一操作接口；多个标准网络协议栈 TCP/IP、HTTP、MQTT、WebSocket 等等；
2. 开发工具集，主要包含用于代码编译的 GCC 交叉工具链、基于 OpenOCD 的 JTAG 调试工具、基于 Segger SystemView 的实时跟踪、Flash 和 eFuse 编程器等；
3. 开发指导文档，详尽的说明了乐鑫各个芯片平台、各个软件版本下的 API 参考，使用指引和注意事项等，开发者可[在线浏览](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/)也可[下载到本地](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/esp-idf-en-v4.4-dev-3540-g4e03a9c-esp32.pdf)查看。
### 框架层 Framework Layer
除了系统层支持外，ESP-BOX 还集成了多个出色的解决方案框架，分别处理不同功能场景下的典型问题。包括用于处理语音唤醒词和命令词的语音识别框架 ESP-SR，处理显示和触摸交互功能的人机交互框架 ESP-HMI，和处理物联网功能的端到端物联网开发框架 ESP-RainMaker，接下来将简要介绍这些解决方案框架的功能和定制化方法：
**ESP-SR**
ESP-SR 是乐鑫面向智能语音领域开发的语音识别框架，该框架可利用 ESP32-S3 AI 扩展指令进行运算加速，为开发者提供开箱即用的高性能、高可靠性的语音识别功能，它包括：
1. 声学前端（[Audio Front-End, AFE](https://github.com/espressif/esp-sr/tree/master/acoustic_algorithm)）算法集，用于提升智能语音设备在远场噪声环境中的交互能力，使开发者获得高质量且稳定的音频数据。它主要包括可以有效滤除扬声器回声的回声消除算法（Acoustic Echo Cancellation，AEC）, 用于强化麦克风阵列方向性的盲源分离算法（Blind Source Separation，BSS），和用于处理环境噪声的噪声抑制算法（Noise Suppression，NS）。ESP-SR 将以上算法封装成简单的 API，开发者无需关心这些算法的具体细节，只需要针对具体的应用场景，对需要使用的算法排列组合，并按照 API 的格式要求输入待处理的语音数据，即可得到声学前端算法的处理结果。
2. 唤醒词识别模型（[WakeNet](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/README.md)），用于在连续的音频流中实时监测出特定的语言片段，将智能设备从待机状态唤醒至语音交互状态。WakeNet 通过计算音频片段的梅尔倒频谱（Mel-Frequency Cepstrum, MFC）作为输入特征，使用针对 ESP32-S3 优化的神经网络算法对特征信号进行分类，在监测到命令词时触发唤醒信号，可在噪声环境下获得不小于 80% 的[有效识别率](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/README.md#performance-test)。ESP-BOX 提供了开箱即用的离线唤醒词 “Hi, ESP” 和 “Hi, 乐鑫”，开发者无需额外的开发工作即可使用，同时乐鑫也支持为开发者[定制独特的唤醒词](https://github.com/espressif/esp-sr/blob/master/wake_word_engine/ESP_Wake_Words_Customization.md)。
3. 命令词识别模型（[MultiNet](https://github.com/espressif/esp-sr/blob/master/speech_command_recognition/README.md)），用于在设备唤醒后，识别用户特定的中英文语音命令，例如 “Turn on the air conditioner”，“关闭电灯” 等。MultiNet 通过基于 CRNN 和 CTC 的卷积循环神经网络，对输入音频片段进行多命令词识别，可同时支持最多 100 个离线命令词。 开发者不需要了解识别模型细节，也无需重新进行模型训练，通过简单配置[中文拼音字符串](https://github.com/espressif/esp-sr/blob/master/speech_command_recognition/README.md#chinese-speech-command-recognition)或[英文音标字符串](https://github.com/espressif/esp-sr/blob/master/speech_command_recognition/README.md#english-speech-command-recognition)以及对应 ID，即可添加或修改命令词，模型在识别到语音命令词后将触发包含 ID 信息的事件。
**ESP-HMI**
ESP-HMI 是乐鑫基于开源 GUI 框架 LVGL 设计的人机交互方案，借助 ESP32-S3 强大的计算能力和外设资源，可以实现出色的数据可视化、触摸控制等功能，它包括：
1. 屏幕驱动库：兼容多款主流 8080、SPI、I2C 接口彩色屏幕、单色屏、触摸屏，提供统一的显示驱动接口。开发者可以直接使用已适配的屏幕控制器，也可根据显示驱动接口添加自己的屏幕，UI 界面开发完成可无缝部署到多种屏幕。
2. 开源 UI 模板：乐鑫设计了适用于物联网控制面板、多媒体播放器、语音助手等不同物联网场景的多套开源 UI 模板，帮助开发者进行更专业的二次开发。
3. 硬件参考设计：提供屏幕应用相关的硬件参考设计、电路原理图和开发指南，帮助开发者快速完成硬件设计。
**ESP-RainMaker**
[ESP-RainMaker](https://rainmaker.espressif.com/docs/get-started.html) 是一个完整的物联网开发框架，作为一个端到端平台，可以为开发者提供设备配网、连云、OTA 等一站式的物联网产品功能和示例。它包含：
1. 设备端 SDK，开发者通过设计设备参数和属性，即可使用乐鑫芯片定义自己的物联网产品，快速完成设备端固件开发;
2. 透明的云中间件，开发者无需管理设备证书和云端基础设施，直接使用乐鑫芯片即可快速建立与云端的建立安全的通信链接；
3. iOS / Android 平台 APP，集成了设备发现，Wi-Fi 配网，用户登录，设备关联和控制等必要功能，开发者无需进行额外的代码开发工作。
**AI 指令集和库**
借助 ESP32-S3 Xtensa® 32­bit LX7 双核 CPU 和 AI / DSP 扩展指令集，ESP32-S3 可高效完成向量运算、复数运算、FFT 等运算，加速神经网络计算和数字信号处理计算能力。AI 开发者们可以基于乐鑫开源的基于 AI / DSP 扩展指令集的软件库，实现高性能的图像识别、语音唤醒与识别等应用。
**设备驱动 Device Drivers**
### 应用层 Application Layer