Claude-2 100k model never outputs more than 4k tokens
Problem
Even though cloude-2 has 100k tokens context window I can not get it to generate more than 4k tokens. It interrupts generation after exactly 4096 tokens. Even if I set `max_tokens_to_sample` to be more than 4096. I tried both anthropic-sdk-python and web interface - both interrupt in the same place. Code to reproduce: [code block] If you run that code you can see, that completion interrupted mid-sentense after reaching 4096 tokens: [code block] It is the last chars of the output. And in developer log on anthropic page it looks like that: <img width="839" alt="Screenshot 2023-09-15 at 11 00 37" src="https://github.com/anthropics/anthropic-sdk-python/assets/120242470/6be42ad7-4038-41eb-9b4d-67c5d8aaa3d4"> So that 100k tokens only for input, and only 4k of them for output? It is very unexpected (because openai models not like that) and not documented behaviour.
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Solution: Claude-2 100k model never outputs more than 4k tokens
@akhil4rajan To overcome the issue I have switched to using openai gpt-4-turbo with 128k context size and without additional output size limitations.
Trust Score
2 verifications
- 1
@akhil4rajan
To overcome the issue I have switched to using openai gpt-4-turbo with 128k context size and without additional output size limitations.
Validation
Resolved in anthropics/anthropic-sdk-python GitHub issue #157. Community reactions: 1 upvotes.
Verification Summary
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep