FG
🤖 AI & LLMs

How to get token count, callbacks work but just for ChatOpenAI but not for RetrievalQAChain

Freshabout 21 hours ago
Mar 14, 20260 views
Confidence Score59%
59%

Problem

I am trying to get a token count for a process, I am passing callbacks to the class initialization like this [code block] However all of the calls from a RetrievalQAChain end up in the catch portion of that try-catch block as 'tokenUsage' does not exist for those calls. Can someone point me in the right direction?

Unverified for your environment

Select your OS to check compatibility.

1 Fix

Canonical Fix
Unverified Fix
New Fix – Awaiting Verification

Implement Token Count Retrieval for RetrievalQAChain

Medium Risk

The RetrievalQAChain does not inherently support token usage tracking as it is designed differently from the ChatOpenAI class. The callbacks that track token usage are likely not being invoked during the execution of the RetrievalQAChain, leading to the 'tokenUsage' not being available in the catch block.

Awaiting Verification

Be the first to verify this fix

  1. 1

    Extend RetrievalQAChain to Support Callbacks

    Modify the RetrievalQAChain class to include a mechanism for tracking token usage. This can be done by adding a callback parameter that can log token usage during the query execution.

    python
    class CustomRetrievalQAChain(RetrievalQAChain):
    def __init__(self, *args, token_callback=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.token_callback = token_callback

    def _call(self, inputs):
        # Call the original method
        result = super()._call(inputs)
        # Assuming 'token_count' is available after the call
        if self.token_callback:
            self.token_callback(token_count)
        return result
  2. 2

    Create Token Count Callback Function

    Define a callback function that will handle the token count and log it accordingly. This function should be passed to the CustomRetrievalQAChain during initialization.

    python
    def log_token_count(token_count):
    print(f'Token count used: {token_count}')
  3. 3

    Initialize Custom RetrievalQAChain with Callback

    When creating an instance of the CustomRetrievalQAChain, pass the log_token_count function as the token_callback parameter to ensure token usage is tracked.

    python
    retrieval_chain = CustomRetrievalQAChain(..., token_callback=log_token_count)
  4. 4

    Test the Implementation

    Run a test query through the CustomRetrievalQAChain to ensure that the token count is being logged correctly and that the process does not throw errors.

    python
    result = retrieval_chain({'query': 'Your test query here'})

Validation

Confirm that the token count is printed to the console during the execution of the test query. Additionally, check that no errors are thrown and that the expected results are returned from the chain.

Sign in to verify this fix

Environment

Submitted by

AC

Alex Chen

2450 rep

Tags

langchainllmairag