Japanese tokenizer library

tokenizer-japanese wraps Kuromoji IPAdic behind JapaneseProcessor. It provides morphological tokenization, POS predicates and filters, compound-aware blockword detection, masking, and runtime dictionary management.

What it provides

• Kuromoji IPAdic tokenization into TokenBase values;
• filterNoun and predicate-based filter;
• isNoun, isVerb, isNounOrVerb, isAdjective, isJosa, and isPunctuation helpers;
• noun/verb blockword detection, including adjacent compound checks;
• add, remove, and clear operations for the in-memory blockword dictionary.

Add the dependency

dependencies {
    implementation("io.github.bluetape4k.text:tokenizer-japanese:0.2.1")
}

Smallest useful example

import io.bluetape4k.tokenizer.japanese.JapaneseProcessor

val tokens = JapaneseProcessor.tokenize("お寿司が食べたい。")
println(tokens.map { it.surface })
// [お, 寿司, が, 食べ, たい, 。]

The returned tokens retain dictionary and POS details. Use surface only when the displayed form is enough.

Filter by part of speech

import io.bluetape4k.tokenizer.japanese.tokenizer.isVerb

val analyzed = JapaneseProcessor.tokenize("私は、日本語の勉強をしています。")
val nouns = JapaneseProcessor.filterNoun(analyzed).map { it.surface }
val verbs = JapaneseProcessor.filter(analyzed) { it.isVerb() }.map { it.surface }

println(nouns) // [私, 日本語, 勉強]
println(verbs) // [し]

POS filtering is morphology-aware. It is more appropriate than raw substring search when the policy depends on grammatical category.

Detect and mask blockwords

import io.bluetape4k.tokenizer.model.blockwordRequestOf

val response = JapaneseProcessor.maskBlockwords(
    blockwordRequestOf("ホモの男性を理解できない"),
)

println(response.maskedText)       // **の男性を理解できない
println(response.blockwordExists)  // true

The built-in path targets noun and verb tokens and also checks adjacent noun plus noun/verb compounds when a single token does not match.

Dictionary lifetime

The packaged dictionary loads lazily on first blockword access. addBlockwords, removeBlockwords, and clearBlockwords mutate the process-wide in-memory policy. Warm the first access when latency matters, and restore application updates after restart.

When to choose it

Choose this module when Japanese token boundaries or POS matter. Choose text-search for exact patterns independent of morphology. Route unknown-language input through Lingua only when routing is required.

Constraints and failure behavior

The facade enforces core request length limits. Kuromoji dictionary loading and the first blockword dictionary access contribute startup work. Treat processor failures as sanitized internal errors rather than returning the original text.

Continue learning

• Mixed-language processing
• Tokenizer safety example
• Startup and memory

Source evidence

• JapaneseProcessor
• Japanese module README
• Japanese processor tests

Release diagrams

These diagrams are loaded directly from README assets published with the 0.2.1 release and pinned to its immutable commit. They describe this manual’s released structure and runtime flows, not later Snapshot changes. Select a preview to open the SVG at the same release commit.

tokenizer japanese Class Structure diagram

Release README: tokenizer-japanese/README.md