BruceLM - Running LLMs locally in Bruce on esp32 devices

[Doominator1]

Proposed Changes

Adding ability to run LLM models in bruce.
Fully local LM running right on the esp32 in Bruce firmware (s3 recommended)
Inference code based off llama2.c by Karpathy

Types of Changes

Feature addition

Verification

1. Build
2. Flash
3. Navigate to the "others" selection in the main menu
4. Scroll down and select "BruceLM"
5. The BruceLM module should open
6. select the model files & tokeniser (you must get both model files from sd_files/BruceLM/models seperately and place them on your sd card in BruceLM/models/stories260K.bin + tok512.bin)
7. Send a prompt and read the model's response

Testing

This was tested on the esp32-s3 based t-embed cc1101 plus.
I have not tested on any other devices, but I assume it will work on all s3 based devices
I have not tested on other esp varient devices such as the c5 or others

User-Facing Change

Added BruceLM: run fully-local LLM models directly on-device (ESP32-S3 recommended), based on llama2.c. Accessible via Others > BruceLM.

Further Comments

Due to the esp32-s3's limitations, only extremely small models are possible to run.
It runs the stories260K model at about ~20 tokens/sec on my esp32-s3 t-embed cc1101 plus
Models are a .bin file inside sdcard/BruceLM/models/ and require a tokenizer .bin as well in same dir

I have provided two compatable model files and their tokeniser files into sd_files

I trained my own chat(-ish) model that can run on bruceLM, with similar architecture as Stories260K. and have added a download link to both compatable models here: https://archive.org/details/BruceLM

[Doominator1]

here is a flashable .bin if u want to try it out or test it on the t-embed cc1101 / t-embed plus
LMBruce-lilygo-t-embed-cc1101.bin.zip
edit - update .bin for latest changes

[Doominator1]

Drafted for now because im making a custom built model and training it from scratch using llama2.c!
Really exciting!
Also I must fix some small chat-style inference bugs.

[Hat798]

can you pre-build for esp32-s3

[Doominator1]

can you pre-build for esp32-s3

I'm away from my desktop currently so I can't until next week but if you know how you can compile from source

[Doominator1]

I have finally squashed all bugs I could find, I think this PR is ready for merge now!
I'm going to un-draft this PR now, let's progress!

[Doominator1]

can you pre-build for esp32-s3

What device are you using, I can build it if you tell me what device to build for, or is it just a generic esp32 s3 devkit board?

[Hat798]

@Doominator1 I using esp32-s3-n16r8

[Doominator1]

@Doominator1 I using esp32-s3-n16r8

with a screen? are you actually using a normal device? or a headless devkit? I can't build if I don't know what your running.

[Hat798]

with screen like smochie board

[Doominator1]

with screen like smochie board

so what exact device name to i target the build for? read the list in platformio.ini and tell me what one