HNSWLib-node not found when using in a AWS Lambda function
Problem
I recently made a simple Typescript function to create a VectorStore using HNSWLib-node. It saves the vector store in a folder and then, in another script file, I load and execute a RetrievalQAChain using OpenAI. Everything was working fine until I decided to put that in a AWS Lambda Function. My package.json has the following dependencies: [code block] Also, I double checked and the hnswlib-node folder is inside "node_modules" folder in my lambda function folder. However, I keep getting the following error (from CloudWatch Logs): [code block] Also, this error is not thrown on importing HNSWLib, but only in the following line of code: [code block] This is my import: `const { HNSWLib } = require("langchain/vectorstores/hnswlib")` It seems I'm not the only one with this problem. See this post Tried this and didn't work. Expeted behavior: code would be executed properly, just like when executed on my local machine. Actual behavior: the error pasted above.
Error Output
error (from CloudWatch Logs):
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Resolve HNSWLib-node Not Found in AWS Lambda Function
AWS Lambda has a specific execution environment that may not include all native dependencies or may not handle certain file structures correctly. The error likely arises from the way HNSWLib-node is packaged or how the Lambda function accesses the node_modules directory, especially if there are native binaries involved that are not compatible with the Lambda execution environment.
Awaiting Verification
Be the first to verify this fix
- 1
Ensure Correct Packaging of Dependencies
When deploying to AWS Lambda, ensure that all dependencies, especially those with native binaries like HNSWLib-node, are packaged correctly. Use the AWS Lambda Layers feature or package the node_modules directory in a zip file that includes the correct architecture.
bashzip -r function.zip . - 2
Use AWS Lambda Layers for Native Dependencies
Create a Lambda Layer that includes the HNSWLib-node package. This allows you to separate your function code from the dependencies, ensuring that all native binaries are compatible with the Lambda environment.
bashaws lambda publish-layer-version --layer-name hnswlib-node --zip-file fileb://layer.zip --compatible-runtimes nodejs14.x - 3
Update Lambda Function to Include Layer
After creating the layer, update your Lambda function configuration to include the newly created layer. This ensures that the function has access to the HNSWLib-node package during execution.
bashaws lambda update-function-configuration --function-name YourFunctionName --layers arn:aws:lambda:region:account-id:layer:hnswlib-node:version - 4
Verify Node.js Version Compatibility
Ensure that the Node.js version specified in your Lambda function matches the version used when developing your application. Mismatched versions can lead to compatibility issues with native modules.
bashCheck the runtime setting in the AWS Lambda console. - 5
Test the Lambda Function
Deploy the changes and invoke the Lambda function to verify that the HNSWLib-node can be imported and used without errors. Monitor the CloudWatch logs for any further issues.
bashaws lambda invoke --function-name YourFunctionName output.txt
Validation
Confirm that the Lambda function executes successfully without throwing the 'HNSWLib-node not found' error. Check the output and CloudWatch logs for any remaining issues.
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep