ModuleNotFoundError: No module named 'tokenizers.tokenizers'
Problem
While using the latest version of anthropic, I'm seeing the above error. I used traceback and its root cause seems to be the tokenizer package: `line 19, in <module>\n from anthropic import AuthenticationError, NotFoundError\n File \"/opt/python/anthropic/__init__.py\", line 6, in <module>\n from ._client import (\n File \"/opt/python/anthropic/_client.py\", line 10, in <module>\n from tokenizers import Tokenizer type: ignore[import]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/python/tokenizers/__init__.py\", line 78, in <module>\n from .tokenizers import (\nModuleNotFoundError: No module named 'tokenizers.tokenizers'\n` Any ideas on how to resolve this? I also opened an issue on tokenizers library: https://github.com/huggingface/tokenizers/issues/1416 Installed: `anthropic==0.7.8`, `tokenizers==0.15.0`
Error Output
Error: No module named 'tokenizers.tokenizers'\n`
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Fix ModuleNotFoundError for tokenizers in Anthropic
The error occurs because the 'tokenizers' package is not correctly installed or the version installed does not contain the 'tokenizers' module. The 'anthropic' library expects a specific structure in the 'tokenizers' package that may not be present in the installed version.
Awaiting Verification
Be the first to verify this fix
- 1
Uninstall existing tokenizers package
Remove the currently installed version of the 'tokenizers' package to avoid conflicts.
bashpip uninstall tokenizers -y - 2
Install compatible version of tokenizers
Install a version of the 'tokenizers' package that is compatible with the 'anthropic' library. As of the latest information, version 0.13.2 is known to work well.
bashpip install tokenizers==0.13.2 - 3
Verify installation
Check that the 'tokenizers' package is correctly installed and contains the required modules.
bashpython -c "import tokenizers; print(dir(tokenizers))" - 4
Run your application
After reinstalling the correct version of 'tokenizers', run your application again to see if the error persists.
bashpython your_application.py
Validation
To confirm the fix worked, ensure that the application runs without raising the 'ModuleNotFoundError' for 'tokenizers.tokenizers'. Additionally, the output of the verification step should list the expected modules within 'tokenizers'.
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep