Tokenizer Error
Problem
When I run: pip install anthropic Then: from anthropic import Anthropic client = Anthropic() client.count_tokens('Hello world!') 3 I get the error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".pyenv\pyenv-win\versions\3.11.3\Lib\site-packages\anthropic\_client.py", line 225, in count_tokens tokenizer = self.get_tokenizer() ^^^^^^^^^^^^^^^^^^^^ File ".pyenv\pyenv-win\versions\3.11.3\Lib\site-packages\anthropic\_client.py", line 230, in get_tokenizer return sync_get_tokenizer() ^^^^^^^^^^^^^^^^^^^^ File ".pyenv\pyenv-win\versions\3.11.3\Lib\site-packages\anthropic\_tokenizers.py", line 33, in sync_get_tokenizer text = tokenizer_path.read_text() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".pyenv\pyenv-win\versions\3.11.3\Lib\pathlib.py", line 1059, in read_text return f.read() ^^^^^^^^ File ".pyenv\pyenv-win\versions\3.11.3\Lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1980: character maps to <undefined> I'm on a windows machine and setting my environment variables -> system settings PYTHONUTF8 to 1 didn't work either.
Error Output
error: Traceback (most recent call last):
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Solution: Tokenizer Error
Thanks for the report @williambarberjr ! We'll take a look at this soon.
Trust Score
1 verification
- 1
Thanks for the report @williambarberjr ! We'll take a look at this soon.
Thanks for the report @williambarberjr ! We'll take a look at this soon.
Validation
Resolved in anthropics/anthropic-sdk-python GitHub issue #132. Community reactions: 0 upvotes.
Verification Summary
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep