You're right for many applications. Yet, for many other applications, simply converting each document into an embedding, converting the search string into an embedding, and doing a search (e.g. cosine similarity), is all that is needed.
My first attempt at RAG was just that, and I was blown away at how effective it was. It worked primarily because each document was short enough (a few lines of text) that you didn't lose much by making an embedding for the whole document.
My next attempt failed, for the reasons you mention.
Point being: Don't be afraid to give the simple/trivial approach a try.
Most people need it just for a demo, though. I'm not developing any applications for it, but I have several ideas in mind for RAG just for my personal use case on my local machine.
And for the one that worked, BTW, would work almost unchanged in an application. I may put it online one day as several people have expressed interest - simply because there are no existing sites that do the job as well as my POC. The challenges will mostly be the usual web app challenges (rate limits, etc), but the actual RAG component doesn't need improvement to be the "best" one out there.
You're right for many applications. Yet, for many other applications, simply converting each document into an embedding, converting the search string into an embedding, and doing a search (e.g. cosine similarity), is all that is needed.
My first attempt at RAG was just that, and I was blown away at how effective it was. It worked primarily because each document was short enough (a few lines of text) that you didn't lose much by making an embedding for the whole document.
My next attempt failed, for the reasons you mention.
Point being: Don't be afraid to give the simple/trivial approach a try.