OpenCVForUnity.DnnModule.Tokenizer Class Reference

High-level tokenizer wrapper for DNN usage. More...

Public Member Functions
string	decode (MatOfInt tokens)
MatOfInt	encode (string text)
	Encode UTF-8 text to token ids (special tokens currently disabled).
IntPtr	getNativeObjAddr ()
Public Member Functions inherited from OpenCVForUnity.DisposableObject
void	Dispose ()
	Releases resources used by this object.
void	ThrowIfDisposed ()
	Throws ObjectDisposedException if this object has been disposed.

Static Public Member Functions
static Tokenizer	__fromPtr__ (IntPtr addr)
static Tokenizer	load (string model_config)
	Load a tokenizer from a model directory.
Static Public Member Functions inherited from OpenCVForUnity.DisposableObject
static IntPtr	ThrowIfNullIntPtr (IntPtr ptr)
	Returns the native pointer, or throws CoreModule.CvException if it is zero.

Protected Member Functions
override void	Dispose (bool disposing)
Protected Member Functions inherited from OpenCVForUnity.DisposableOpenCVObject
	DisposableOpenCVObject ()
	Initializes a new instance with a zero native pointer and dispose enabled.
	DisposableOpenCVObject (bool isEnabledDispose)
	Initializes a new instance with a zero native pointer.
	DisposableOpenCVObject (IntPtr ptr)
	Initializes a new instance with the specified native pointer and dispose enabled.
	DisposableOpenCVObject (IntPtr ptr, bool isEnabledDispose)
	Initializes a new instance with the specified native pointer.
Protected Member Functions inherited from OpenCVForUnity.DisposableObject
	DisposableObject ()
	Initializes a new instance with dispose enabled.
	DisposableObject (bool isEnabledDispose)
	Initializes a new instance.

Additional Inherited Members
Package Attributes inherited from OpenCVForUnity.DisposableOpenCVObject
Properties inherited from OpenCVForUnity.DisposableObject
bool	IsDisposed [get, protected set]
bool	IsEnabledDispose [get, set]

Detailed Description

High-level tokenizer wrapper for DNN usage.

Provides a simple API to encode and decode tokens for LLMs. Models are loaded via Tokenizer.load().

using namespace cv::dnn;
Tokenizer tok = Tokenizer::load("/path/to/model/");
std::vector<int> ids = tok.encode("hello world");
std::string text = tok.decode(ids);

Member Function Documentation

__fromPtr__()

static Tokenizer OpenCVForUnity.DnnModule.Tokenizer.__fromPtr__ ( IntPtr addr )

static

decode()

string OpenCVForUnity.DnnModule.Tokenizer.decode

(

MatOfInt

tokens

)

Dispose()

override void OpenCVForUnity.DnnModule.Tokenizer.Dispose ( bool disposing )

protectedvirtual

Reimplemented from OpenCVForUnity.DisposableOpenCVObject.

encode()

MatOfInt OpenCVForUnity.DnnModule.Tokenizer.encode

(

string

text

)

Encode UTF-8 text to token ids (special tokens currently disabled).

Calls the underlying CoreBPE::encode with an empty allowed-special set.

Parameters

text	UTF-8 input string.

Returns

Vector of token ids (32-bit ids narrowed to int for convenience).

getNativeObjAddr()

IntPtr OpenCVForUnity.DnnModule.Tokenizer.getNativeObjAddr

(

)

load()

static Tokenizer OpenCVForUnity.DnnModule.Tokenizer.load ( string model_config )

static

Load a tokenizer from a model directory.

Expects the directory to contain:

• config.json with field model_type with value "gpt2" or "gpt4".
• tokenizer.json produced by the corresponding model family.

The argument is a path prefix; this function concatenates file names directly (e.g. model_dir + "config.json"), so model_dir must end with an appropriate path separator.

Parameters

model_config

Path to config.json for model.

Returns

A Tokenizer ready for use. Throws cv::Exception if files are missing or model_type is unsupported.

---

The documentation for this class was generated from the following file:

• OpenCVForUnity/Assets/OpenCVForUnity/Scripts/Core/org/opencv/dnn/Tokenizer.cs