Increase max vectors dimension limit for index
Problem
Hello ! I'm wondering what are the reasons for the maximal dimension limit for indexes HNSW and IVFFlat. Checking _really quickly_ the code, I see those `HNSW_MAX_DIM` and `IVFFLAT_MAX_DIM` constants set to 2000 which seem arbitrary to me, as a non-educated user. - Why do we have these limits ? - How would performance scale if the limited were, say, doubled ? - Is there a plan to increase the limits, considering other vector database products offer up to 32768 dimensions (or even more) for indexed vector ? Thank you !
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Solution: Increase max vectors dimension limit for index
It would be very interesting to be able to increase this limit beyond 2000. The new mistral-derivative embedding models (4096 dims) and OpenAI’s new text-embedding-3-large model (3072 dims) offer very good performances, but are not yet compatible with pgvector. We're likely to see more and more good models with an increasing number of dimensions.
Trust Score
6 verifications
- 1
It would be very interesting to be able to increase this limit beyond 2000. The
It would be very interesting to be able to increase this limit beyond 2000. The new mistral-derivative embedding models (4096 dims) and OpenAI’s new text-embedding-3-large model (3072 dims) offer very good performances, but are not yet compatible with pgvector. We're likely to see more and more good models with an increasing number of dimensions.
Validation
Resolved in pgvector/pgvector GitHub issue #461. Community reactions: 19 upvotes.
Verification Summary
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep