FG
💻 Software🤖 AI & LLMsAnthropic

Tokenizer Error

Freshover 2 years ago
Mar 14, 20260 views
Confidence Score86%
86%

Problem

When I run: pip install anthropic Then: from anthropic import Anthropic client = Anthropic() client.count_tokens('Hello world!') 3 I get the error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".pyenv\pyenv-win\versions\3.11.3\Lib\site-packages\anthropic\_client.py", line 225, in count_tokens tokenizer = self.get_tokenizer() ^^^^^^^^^^^^^^^^^^^^ File ".pyenv\pyenv-win\versions\3.11.3\Lib\site-packages\anthropic\_client.py", line 230, in get_tokenizer return sync_get_tokenizer() ^^^^^^^^^^^^^^^^^^^^ File ".pyenv\pyenv-win\versions\3.11.3\Lib\site-packages\anthropic\_tokenizers.py", line 33, in sync_get_tokenizer text = tokenizer_path.read_text() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".pyenv\pyenv-win\versions\3.11.3\Lib\pathlib.py", line 1059, in read_text return f.read() ^^^^^^^^ File ".pyenv\pyenv-win\versions\3.11.3\Lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1980: character maps to <undefined> I'm on a windows machine and setting my environment variables -> system settings PYTHONUTF8 to 1 didn't work either.

Error Output

error:
Traceback (most recent call last):

Unverified for your environment

Select your OS to check compatibility.

1 Fix

Canonical Fix
Moderate Confidence Fix
84% confidence100% success rate1 verificationLast verified Mar 14, 2026

Solution: Tokenizer Error

Low Risk

Thanks for the report @williambarberjr ! We'll take a look at this soon.

84

Trust Score

1 verification

100% success
  1. 1

    Thanks for the report @williambarberjr ! We'll take a look at this soon.

    Thanks for the report @williambarberjr ! We'll take a look at this soon.

Validation

Resolved in anthropics/anthropic-sdk-python GitHub issue #132. Community reactions: 0 upvotes.

Verification Summary

Worked: 1
Last verified Mar 14, 2026

Sign in to verify this fix

Environment

Submitted by

AC

Alex Chen

2450 rep

Tags

claudeanthropicllmapi