Initial Chunking Subnet Benchmarks:
The Power of Open, Incentivized Contest
Completed Sep 5, 2024

Check out our notebook for a closer look at our methodology and how to query the top miners.

🔗 Juptyer Notebook: https://t.co/tkufM4haT4
🔗 GitHub Repo: https://t.co/62Cqes7sDl

We’re excited to share that we've released the first benchmarks for the Chunking Subnet!

This benchmark considers the chunked response of the top-ranked miner, against the industry leaders @AI21Labs and @UnstructuredIO, on a fiction RAG benchmark.

With Top K = 5 (meaning the five most relevant chunks included as context), the @chunking_subnet scored 61.9%, @UnstructuredIO 56.7% and @AI21Labs 45.7%.

This benchmark will be taken weekly, and soon visualized on our subnet dashboard: https://t.co/T201iuXMaz

Note: The subnet is still in its early stages, but is starting to take shape! The next patch will have validators pseudo-randomly select max chunk size and max chunk quantity, so that miners must optimize for a range of common applications.