Please add a note to README warning of limitations on Windows
Problem
On Windows, `maintenance_work_mem` cannot be bigger than 2GB, something that is poorly documented. Because of this, building indices is prohibitively slow for larger vector stores. In order for others not to have to find this out the hard way like I did, I propose adding a small note to the README that warns people of this. Maybe something like this: [code block] Documenting my experience: I created 37 million vectors (PubMed) of 384 dimensions. Loading them in pgvector using bulk loading was quick (< 2 hours). But when I run [code block] After running for 65 hours, progress is at 74.5%, which is 1% higher than it was 24 hours ago.
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Add README Note on Windows Memory Limitation for pgvector
On Windows systems, the PostgreSQL configuration parameter `maintenance_work_mem` is limited to a maximum of 2GB. This limitation is not well-documented, leading to significant performance issues when building indices for large vector stores, as users may unknowingly set this parameter higher, resulting in extremely slow processing times.
Awaiting Verification
Be the first to verify this fix
- 1
Identify README Location
Locate the README file in the project repository where user instructions and configurations are provided.
- 2
Draft Warning Note
Create a warning note that clearly states the limitation of `maintenance_work_mem` on Windows. The note should inform users that setting this parameter above 2GB will not have any effect and may lead to performance degradation.
markdown### Important Note for Windows Users On Windows, the `maintenance_work_mem` setting cannot exceed 2GB. Setting this value higher will not improve performance and may lead to significantly slower index building times for large vector stores. - 3
Insert Note into README
Insert the drafted warning note into the appropriate section of the README file, ideally near the configuration settings or usage instructions for pgvector.
- 4
Review and Test Changes
Review the changes made to the README for clarity and correctness. Test the README formatting to ensure the note is displayed properly in the rendered view on platforms like GitHub.
- 5
Commit Changes
Commit the changes to the repository with a clear message indicating that a warning note regarding Windows memory limitations has been added to the README.
bashgit commit -m 'Add warning note about Windows memory limitation for pgvector'
Validation
Confirm that the README file displays the new warning note correctly and that users are aware of the `maintenance_work_mem` limitation when using pgvector on Windows. Monitor feedback from users to ensure the note is helpful.
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep