I got interested in vector search around 2004, read a lot of papers about vector search algorithms and was not really impressed with the tradeoffs involved (it's not the clear win that B-Trees are for 1-d indexing) and wound up using full scans unless I had sparse vectors.
When Pinecone came out and started blogging heavily it seemed that they'd read the same papers I did but came to the conclusion the glass was half full instead of half empty. I could have missed it but I haven't see anything in the literature that's a huge improvement over 20 year old algos.
Circa 2014 I worked on a search engine for patents and related literature that made vectors for 20 million + documents and they decided to use full scan and (i) it performed so well (in terms of accuracy) that we sold a license to the USPTO on day two after we put up the demo, and (ii) there were a lot of things about it that were slow like the build system, index building and model training but vector search wasn't one of them.
My YOShInOn RSS reader has about a million documents in 2024 and it uses vectors for classification and clustering. Using vectors for search is a clear extension and I've done some prototyping of searches with full-scan and performance is "good enough" (full scan has 'mechanical sympathy'.) I'd probably stuff my vectors into FAISS if I wanted to do anything more and forget about it.
Sending my vectors to some cloud service so they can pay AWS prices to store them? That's for the birds. I respect Pinecone for being early to the party but I think those who jumped in in 2022 were laggards.
My experience comes from around the same time frame -- I spent about a year on an aborted spectral dimension reduction project, and I only recently realized how similar the problem still is today.
I'm not sure if that makes me more or less qualified to do vector DB's -- I tend to block out things that I learned a lot about in the past without much result.
When Pinecone came out and started blogging heavily it seemed that they'd read the same papers I did but came to the conclusion the glass was half full instead of half empty. I could have missed it but I haven't see anything in the literature that's a huge improvement over 20 year old algos.
Circa 2014 I worked on a search engine for patents and related literature that made vectors for 20 million + documents and they decided to use full scan and (i) it performed so well (in terms of accuracy) that we sold a license to the USPTO on day two after we put up the demo, and (ii) there were a lot of things about it that were slow like the build system, index building and model training but vector search wasn't one of them.
My YOShInOn RSS reader has about a million documents in 2024 and it uses vectors for classification and clustering. Using vectors for search is a clear extension and I've done some prototyping of searches with full-scan and performance is "good enough" (full scan has 'mechanical sympathy'.) I'd probably stuff my vectors into FAISS if I wanted to do anything more and forget about it.
Sending my vectors to some cloud service so they can pay AWS prices to store them? That's for the birds. I respect Pinecone for being early to the party but I think those who jumped in in 2022 were laggards.