com.litongjava.openai.token

Class OpenAiTokenizer

java.lang.Object

com.litongjava.openai.token.OpenAiTokenizer

All Implemented Interfaces:

dev.langchain4j.model.Tokenizer

public class OpenAiTokenizer
extends Object
implements dev.langchain4j.model.Tokenizer

This class can be used to estimate the cost (in tokens) before calling OpenAI or when using streaming. Magic numbers present in this class were found empirically while testing. There are integration tests in place that are making sure that the calculations here are very close to that of OpenAI.

Constructor Summary

Constructors

Constructor and Description
OpenAiTokenizer() Creates an instance of the OpenAiTokenizer for the "gpt-3.5-turbo" model.
OpenAiTokenizer(String modelName) Creates an instance of the OpenAiTokenizer for a given model name.

Method Summary

Modifier and Type	Method and Description
String	decode(List<Integer> tokens)
List<Integer>	encode(String text)
List<Integer>	encode(String text, int maxTokensToEncode)
int	estimateTokenCountInForcefulToolExecutionRequest(dev.langchain4j.agent.tool.ToolExecutionRequest toolExecutionRequest)
int	estimateTokenCountInForcefulToolSpecification(dev.langchain4j.agent.tool.ToolSpecification toolSpecification)
int	estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)
int	estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)
int	estimateTokenCountInText(String text)
int	estimateTokenCountInToolExecutionRequests(Iterable<dev.langchain4j.agent.tool.ToolExecutionRequest> toolExecutionRequests)
int	estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface dev.langchain4j.model.Tokenizer

estimateTokenCountInTools, estimateTokenCountInTools

Constructor Detail

OpenAiTokenizer

public OpenAiTokenizer()

Creates an instance of the OpenAiTokenizer for the "gpt-3.5-turbo" model. It should be suitable for all current OpenAI models, as they all use the same cl100k_base encoding.

OpenAiTokenizer

public OpenAiTokenizer(String modelName)

Creates an instance of the OpenAiTokenizer for a given model name.

Method Detail

estimateTokenCountInText

public int estimateTokenCountInText(String text)

Specified by:

estimateTokenCountInText in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInMessage

public int estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)

Specified by:

estimateTokenCountInMessage in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInMessages

public int estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)

Specified by:

estimateTokenCountInMessages in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInToolSpecifications

public int estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications)

Specified by:

estimateTokenCountInToolSpecifications in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInForcefulToolSpecification

public int estimateTokenCountInForcefulToolSpecification(dev.langchain4j.agent.tool.ToolSpecification toolSpecification)

Specified by:

estimateTokenCountInForcefulToolSpecification in interface dev.langchain4j.model.Tokenizer

encode

public List<Integer> encode(String text)

encode

public List<Integer> encode(String text,
                            int maxTokensToEncode)

decode

public String decode(List<Integer> tokens)

estimateTokenCountInToolExecutionRequests

public int estimateTokenCountInToolExecutionRequests(Iterable<dev.langchain4j.agent.tool.ToolExecutionRequest> toolExecutionRequests)

Specified by:

estimateTokenCountInToolExecutionRequests in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInForcefulToolExecutionRequest

public int estimateTokenCountInForcefulToolExecutionRequest(dev.langchain4j.agent.tool.ToolExecutionRequest toolExecutionRequest)

Specified by:

estimateTokenCountInForcefulToolExecutionRequest in interface dev.langchain4j.model.Tokenizer