The optimal chunking strategy is often highly, highly dependent on the data used... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

thenaturalist 3 months ago | parent | context | favorite | on: Launch HN: Panora (YC S24) – Data Integration API ...

The optimal chunking strategy is often highly, highly dependent on the data used and questions to be answered.

The net is plastered with blog posts about optimal strategies, of which there seem to be more than 10 and new approaches popping up often.

It seems consensus that trial and error is the way to go to optimize cost and performance.

How do you plan to tackle this when providing it out of the box?

nael_ob 3 months ago [–]

That's why we wanted to try the OSS approach where contributors can help keep up with the optimal strategy. We also plan to build an engine to test each strategy and compare retrieval perf before choosing one at runtime.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact