com.litongjava.openai.token

类 OpenAiTokenizer

java.lang.Object

com.litongjava.openai.token.OpenAiTokenizer

所有已实现的接口:

dev.langchain4j.model.Tokenizer

public class OpenAiTokenizer
extends Object
implements dev.langchain4j.model.Tokenizer

This class can be used to estimate the cost (in tokens) before calling OpenAI or when using streaming. Magic numbers present in this class were found empirically while testing. There are integration tests in place that are making sure that the calculations here are very close to that of OpenAI.

构造器概要

构造器

构造器和说明
OpenAiTokenizer() Creates an instance of the OpenAiTokenizer for the "gpt-3.5-turbo" model.
OpenAiTokenizer(String modelName) Creates an instance of the OpenAiTokenizer for a given model name.

方法概要

限定符和类型	方法和说明
String	decode(List<Integer> tokens)
List<Integer>	encode(String text)
List<Integer>	encode(String text, int maxTokensToEncode)
int	estimateTokenCountInForcefulToolExecutionRequest(dev.langchain4j.agent.tool.ToolExecutionRequest toolExecutionRequest)
int	estimateTokenCountInForcefulToolSpecification(dev.langchain4j.agent.tool.ToolSpecification toolSpecification)
int	estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)
int	estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)
int	estimateTokenCountInText(String text)
int	estimateTokenCountInToolExecutionRequests(Iterable<dev.langchain4j.agent.tool.ToolExecutionRequest> toolExecutionRequests)
int	estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications)

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 dev.langchain4j.model.Tokenizer

estimateTokenCountInTools, estimateTokenCountInTools

构造器详细资料

OpenAiTokenizer

public OpenAiTokenizer()

Creates an instance of the OpenAiTokenizer for the "gpt-3.5-turbo" model. It should be suitable for all current OpenAI models, as they all use the same cl100k_base encoding.

OpenAiTokenizer

public OpenAiTokenizer(String modelName)

Creates an instance of the OpenAiTokenizer for a given model name.

方法详细资料

estimateTokenCountInText

public int estimateTokenCountInText(String text)

指定者:

estimateTokenCountInText 在接口中 dev.langchain4j.model.Tokenizer

estimateTokenCountInMessage

public int estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)

指定者:

estimateTokenCountInMessage 在接口中 dev.langchain4j.model.Tokenizer

estimateTokenCountInMessages

public int estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)

指定者:

estimateTokenCountInMessages 在接口中 dev.langchain4j.model.Tokenizer

estimateTokenCountInToolSpecifications

public int estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications)

指定者:

estimateTokenCountInToolSpecifications 在接口中 dev.langchain4j.model.Tokenizer

estimateTokenCountInForcefulToolSpecification

public int estimateTokenCountInForcefulToolSpecification(dev.langchain4j.agent.tool.ToolSpecification toolSpecification)

指定者:

estimateTokenCountInForcefulToolSpecification 在接口中 dev.langchain4j.model.Tokenizer

encode

public List<Integer> encode(String text)

encode

public List<Integer> encode(String text,
                            int maxTokensToEncode)

decode

public String decode(List<Integer> tokens)

estimateTokenCountInToolExecutionRequests

public int estimateTokenCountInToolExecutionRequests(Iterable<dev.langchain4j.agent.tool.ToolExecutionRequest> toolExecutionRequests)

指定者:

estimateTokenCountInToolExecutionRequests 在接口中 dev.langchain4j.model.Tokenizer

estimateTokenCountInForcefulToolExecutionRequest

public int estimateTokenCountInForcefulToolExecutionRequest(dev.langchain4j.agent.tool.ToolExecutionRequest toolExecutionRequest)

指定者:

estimateTokenCountInForcefulToolExecutionRequest 在接口中 dev.langchain4j.model.Tokenizer