Tokenizer core library

tokenizer-core provides shared request and response models, policy options, dictionary loaders, compact character collections, and tokenizer exceptions. Use it directly when you build a boundary or custom processor; applications that only call Korean or Japanese processing normally receive it transitively.

What it provides

• TokenizeRequest, TokenizeResponse, BlockwordRequest, and BlockwordResponse;
• locale and masking options plus Severity.LOW, MIDDLE, and HIGH;
• MAX_TOKENIZE_TEXT_LENGTH and MAX_BLOCKWORD_TEXT_LENGTH, both 100_000 in 0.2.1;
• DictionaryProvider for plain or gzip classpath dictionaries;
• CharArraySet and CharArrayMap for repeated character-sequence lookup;
• TokenizerException and InvalidTokenizeRequestException.

Add the dependency

dependencies {
    implementation("io.github.bluetape4k.text:tokenizer-core:0.2.1")
}

Omit the explicit version when a bluetape4k BOM manages it.

Smallest useful example

import io.bluetape4k.tokenizer.model.TokenizeOptions
import io.bluetape4k.tokenizer.model.tokenizeRequestOf
import java.util.Locale

val request = tokenizeRequestOf(
    text = "코틀린 코루틴",
    options = TokenizeOptions(locale = Locale.KOREAN),
)

println(request.text)
println(request.options.locale)

The factory validates the text before a processor receives it. It also creates the message metadata used by the shared contract.

Dictionary loading

import io.bluetape4k.tokenizer.utils.DictionaryProvider

val words = DictionaryProvider.readWords("dict/base.txt", "dict/custom.txt")
println("blocked" in words)

readWords loads multiple resources through a coroutine-based asynchronous path and returns a CharArraySet. Call it during controlled setup and reuse the result. readWordsAsSequence is the lazy line-oriented alternative; readWordFreqs parses tab-separated word-frequency data.

When to choose it

Use core models at an HTTP or message boundary, for shared dictionary tooling, or when implementing a processor. Choose Korean or Japanese modules when you need actual morphological analysis.

Constraints and failure behavior

Blank text and text longer than 100_000 characters are rejected before processing. Do not include the submitted text in an error response or log. DictionaryProvider expects classpath resources in its supported line formats; resource failures belong to setup and should not be retried on every request.

Continue learning

• Input safety
• Dictionaries and blockwords
• Tokenizer safety example

Source evidence

• TokenizeRequest
• BlockwordRequest
• DictionaryProvider

Release diagrams

These diagrams are loaded directly from README assets published with the 0.2.1 release and pinned to its immutable commit. They describe this manual’s released structure and runtime flows, not later Snapshot changes. Select a preview to open the SVG at the same release commit.

tokenizer core Class Structure diagram

Release README: tokenizer-core/README.md