Langchain doesn't work on AWS Lambda with either faiss or hnswlib
Problem
The current version is broken on AWS lambda, both with Faiss and HNSWlib. I've spent the last few days building test cases on both platforms, but haven't dived into the langchain code. The typical outcome is: "Could not import faiss-node. Please install faiss-node as a dependency with, e.g. `npm install -S faiss-node`.\n\nError: libgomp.so.1: cannot open shared object file: No such file or directory" Same with HNSWlib (substitute hnswlib for faiss in the error above). Also see https://github.com/hwchase17/langchainjs/issues/943 https://github.com/hwchase17/langchainjs/issues/1764 Hope this helps - langchain is amazing, and if it worked on AWS it would be even better :) David
Error Output
Error: libgomp.so.1: cannot open shared object file: No such file or directory"
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Fix AWS Lambda Compatibility for Langchain with Faiss and HNSWlib
The error 'libgomp.so.1: cannot open shared object file' indicates that the required shared library for OpenMP (libgomp) is missing in the AWS Lambda execution environment. This library is necessary for the proper functioning of both Faiss and HNSWlib, which rely on parallel processing capabilities.
Awaiting Verification
Be the first to verify this fix
- 1
Create a Lambda Layer with libgomp
To resolve the missing library issue, create a Lambda Layer that includes the libgomp.so.1 file. This can be done by packaging the library from an Amazon Linux environment, which is compatible with AWS Lambda.
bashmkdir -p layer/lib && cp /usr/lib64/libgomp.so.1 layer/lib/ && zip -r layer.zip layer/ - 2
Upload the Layer to AWS Lambda
After creating the layer zip file, upload it to AWS Lambda using the AWS Management Console or AWS CLI. This layer will provide the necessary shared library for your Lambda function.
bashaws lambda publish-layer-version --layer-name libgomp-layer --zip-file fileb://layer.zip --compatible-runtimes nodejs14.x nodejs16.x - 3
Attach the Layer to Your Lambda Function
Once the layer is uploaded, attach it to your Lambda function. This can be done through the AWS Management Console by navigating to your function's configuration and adding the newly created layer.
bashaws lambda update-function-configuration --function-name your-function-name --layers arn:aws:lambda:your-region:your-account-id:layer:libgomp-layer:1 - 4
Install Faiss and HNSWlib Dependencies
Ensure that both faiss-node and hnswlib are installed as dependencies in your Lambda function. This can be done by including them in your package.json and running npm install.
bashnpm install faiss-node hnswlib - 5
Test the Lambda Function
Invoke your Lambda function to confirm that it can now successfully import faiss-node and hnswlib without errors. Monitor the logs for any issues related to library imports.
bashaws lambda invoke --function-name your-function-name output.txt
Validation
Check the output of the Lambda function invocation. If it runs without the 'libgomp.so.1' error and successfully imports faiss-node and hnswlib, the fix is confirmed.
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep