Package {jiebaRS}

Add user-defined words with an internal native worker.

Description

Internal bridge used by new_user_word() to mutate an existing native worker in place.

Usage

add_user_words(worker, words, tags, freq)

Arguments

Value

NULL, invisibly, after the worker has been updated.

Count n-grams from segmented text

Description

Count contiguous n-grams from a segmented character vector or from each element of a list of segmented character vectors.

This function is a drop-in replacement for jiebaR::get_tuple(), which is deprecated in jiebaRS. See Details for more information.

Usage

count_ngrams(
  x,
  ...,
  n = 2,
  sep = " ",
  sort = TRUE,
  format = c("data.frame", "vector")
)

Arguments

Details

The original jiebaR::get_tuple() interface has several design problems:

1. Its n-gram extraction behavior does not match the most obvious reading of the argument name: size = n counts all contiguous n-grams from 2:n, not just the exact size n.

2. Its documentation says it accepts list input, but the original exported implementation does not reliably support lists.

3. It concatenates tokens without a separator, which makes tuple boundaries ambiguous.

count_ngrams() addresses these issues, providing more explicit and abundant parameters. In addition, this function is about 1.3x to 2.0x faster than jiebaR::get_tuple().

Value

N-gram counts in the requested format.

See Also

get_tuple()

Examples

count_ngrams(c("\u6211", "\u7231", "R"), n = 2)
count_ngrams(c("\u6211", "\u7231", "R"), n = 1:2, format = "data.frame")
count_ngrams(c("a", "b", "b", "b", "a"), n = 1, sort = FALSE)
count_ngrams(list(c("a", "b", "c"), c("a", "b")), n = 2)

Filter segmentation results

Description

Remove selected words from a segmented character vector or from each element of a list of segmented character vectors.

Usage

filter_segment(input, filter_words, keep_na = TRUE)

Arguments

Details

This is a modern reimplementation of jiebaR::filter_segment() with the same core filtering behavior under the default settings.

In the reproducible benchmark, this version is about 110x to 140x faster than jiebaR::filter_segment() on the tested workloads.

Value

An object with the same shape as input, with matching words removed.

Examples

filter_segment(c("abc", "def", " ", "."), c("abc"))
filter_segment(c("a", NA, "b", "a"), c("b"), keep_na = FALSE)
input <- list(
  c("\u6211", "\u662f", "\u6d4b\u8bd5"),
  c("\u6d4b\u8bd5", "\u6587\u672c", "\u6211")
)
filter_segment(input, "\u6211")

The frequency of words

Description

This function returns the frequency of words.

Usage

freq(x, ..., sort = FALSE)

Arguments

Value

A data frame with char and freq columns.

Examples

freq(c("b", "a", "b", "c", "a"))
freq(c("b", "a", "b", "c", "a"), sort = TRUE)

Generate IDF dict

Description

Generate IDF dict from a list of documents.

Usage

get_idf(x, stop_word = NULL, stop_word_file = NULL, path = NULL)

Arguments

Details

Input list contains multiple character vectors with words, and each vector represents a document.

Stop words will be removed from the result.

If path is not NULL, it will write the result to the path.

Value

A data frame with name and count columns, or a file path (invisibly) when path is supplied.

Examples

get_idf(list(c("abc", "def"),c("abc", " ")))

Compatibility wrapper for jiebaR::get_tuple()

Description

get_tuple() is kept only for compatibility with jiebaR. New code should use count_ngrams() instead.

Usage

get_tuple(x, size = 2, dataframe = TRUE)

Arguments

Details

This function is deprecated and should not be used in new code. It is provided only as a compatibility wrapper around count_ngrams() and replicates the behavior of jiebaR::get_tuple().

Prefer count_ngrams() because the original jiebaR::get_tuple() interface has several design problems:

1. Its n-gram extraction behavior does not match the most obvious reading of the argument name: size = n counts all contiguous n-grams from 2:n, not just the exact size n.

2. Its documentation says it accepts list input, but the original exported implementation does not reliably support lists.

3. It concatenates tokens without a separator, which makes tuple boundaries ambiguous.

Value

If dataframe = TRUE, a data frame with name and count columns, sorted by descending count. Otherwise, a named integer vector.

See Also

count_ngrams()

Examples

suppressWarnings(get_tuple(c("sd", "sd", "sd", "rd"), 2))

Extract keywords from text

Description

Extract TF-IDF keywords from a single in-memory string with a keyword worker created by worker(). This is separate from textrank(), which uses TextRank weighting.

Usage

keywords(code, jiebar, ..., format = c("vector", "data.frame", "legacy"))

Arguments

Value

Keyword results in the requested format.

Extract keywords as a data frame

Description

Convenience wrapper around keywords() that always returns a data frame with term and weight columns.

Usage

keywords_df(x, jiebar)

Arguments

Value

A data frame with term and weight columns.

Extract keywords with an internal native worker.

Description

Internal bridge used by keywords() to extract TF-IDF keywords from a single UTF-8 string.

Usage

keywords_worker(text, worker)

Arguments

Value

A named list with keyword and weight vectors.

Add user word

Description

Add one or more custom words to a jieba worker.

Usage

new_user_word(worker, words, tags = "n", freq = NULL)

add_word(worker, words, tags = "n", freq = NULL)

Arguments

Value

NULL, invisibly. Called for its side effect of adding the supplied words to worker, thereby modifying the state used by subsequent operations with the same worker.

Examples

cutter <- worker()
segment("\u91cf\u5b50\u673a\u5668\u72d7", cutter)
new_user_word(cutter, "\u91cf\u5b50\u673a\u5668\u72d7", tags = "n", freq = 1000L)
segment("\u91cf\u5b50\u673a\u5668\u72d7", cutter)

cutter2 <- worker()
add_word(
  cutter2,
  c("\u8d85\u5bfc\u91cf\u5b50\u6bd4\u7279", "\u91cf\u5b50\u673a\u5668\u72d7"),
  tags = c(NA, "n"),
  freq = c(NA, 1000L)
)
segment("\u8d85\u5bfc\u91cf\u5b50\u6bd4\u7279", cutter2)

Create an internal native jieba worker.

Description

Internal bridge used by worker() to allocate a native JiebaWorker.

Usage

new_worker(
  worker_type,
  use_hmm,
  hmm_model,
  idf_path,
  dict_path,
  user_paths,
  top_n,
  min_keyword_length,
  stop_words
)

Arguments

Value

A native JiebaWorker handle.

Segment text with a jieba worker

Description

Segment one or more strings with a jieba_worker created by worker().

Usage

segment(
  code,
  jiebar,
  ...,
  mod = NULL,
  batch = c("list", "data.frame", "flatten")
)

Arguments

Details

For a single input string, segment() always returns a character vector of segmented tokens.

When the worker was created with symbol = TRUE, symbol-like characters are retained during preprocessing. For example, AK-47, N95, X-900, and 4.55 remain intact. With symbol = FALSE (the default), non-letter, non-number, and non-mark characters are replaced with spaces before segmentation, so the same input is returned as AK, 47, N95, X, 900, 4, and 55.

In the current release benchmarks on the bundled Fortress Besieged and Dream of the Red Chamber texts, jiebaRS::segment() is about 1.7x to 1.9x faster than jiebaR::segment() when each novel is segmented as one long string. When the input is many short strings segmented in parallel, jiebaRS::segment() reaches about 7x to 12x speedup over jiebaR.

For very long texts, splitting into about 32 to 128 chunks before segmentation is recommended for good throughput.

For multiple input strings, the argument batch controls how the per-string token vectors are aggregated:

• "list": one character vector per input string.

• "data.frame": a data frame with doc_id and word columns.

• "flatten": all token vectors concatenated into one character vector.

When batch is omitted, jiebaRS returns list output for multi-string input.

The mod argument from jiebaR::segment() is retained only as a deprecated compatibility placeholder. In jiebaRS, segmentation behavior should be controlled by the worker type itself (for example, worker(type = "mix") or worker(type = "query")), not by mutating behavior at call time. When mod is supplied, jiebaRS warns and ignores it.

Value

Segmented tokens in the requested aggregation form.

Examples

seg <- worker()
text1 <- "\u5357\u4eac\u5e02\u957f\u6c5f\u5927\u6865"
text2 <- "\u8fd9\u662f\u4e00\u4e2a\u6d4b\u8bd5"
segment(text1, seg)
segment(c(text1, text2), seg, batch = "list")
segment(c(text1, text2), seg, batch = "data.frame")

Segment a batch of strings

Description

Convenience wrapper around segment() for multi-string input. When batch is omitted, segment_batch() will return list output by default.

Usage

segment_batch(texts, jiebar, ..., batch = c("list", "data.frame", "flatten"))

Arguments

Details

segment_batch() is a convenience wrapper around segment() for explicit batch processing. It always treats texts as multi-string input. The returned object depends on batch:

• "list": one character vector per input string.

• "data.frame": a data frame with doc_id and word columns.

• "flatten": one concatenated character vector.

In the current release benchmarks on the bundled Fortress Besieged and Dream of the Red Chamber texts, batch segmentation reaches about 7x to 12x speedup over the comparable jiebaR workflow on many-string inputs. For very long texts, splitting into about 32 to 128 chunks before calling segment_batch() is recommended for good throughput.

Value

Segmented tokens in the requested aggregation form.

Examples

seg <- worker()
texts <- c("\u5357\u4eac\u5e02\u957f\u6c5f\u5927\u6865", "\u8fd9\u662f\u4e00\u4e2a\u6d4b\u8bd5")
segment_batch(texts, seg)
segment_batch(texts, seg, batch = "flatten")

Segment multiple strings with an internal native worker.

Description

Internal bridge used by segment() to segment many UTF-8 strings.

Usage

segment_batch_worker(texts, worker)

Arguments

Value

A list of character vectors, one per input string.

Segment text with an internal native worker.

Description

Internal bridge used by segment() to segment a single UTF-8 string.

Usage

segment_worker(text, worker)

Arguments

Value

A character vector of segmented tokens.

Tag text with a jiebaRS worker

Description

Tag one or more strings with a jieba_worker created by worker().

Usage

tagging(
  code,
  jiebar,
  ...,
  format = c("vector", "data.frame", "legacy"),
  batch = c("list", "flatten")
)

Arguments

Details

format controls the shape of each single-string tagging result:

• "vector": a named character vector with token names and tag values.

• "data.frame": a data frame with term and tag columns.

• "legacy": the old jiebaR layout with token values and tag names.

In the current release benchmarks on the bundled Fortress Besieged and Dream of the Red Chamber texts, jiebaRS::tagging() is about 1.6x to 1.8x faster than jiebaR::tagging() when each novel is tagged as one long string. When the same content is split into many strings and processed in batch, jiebaRS::tagging() is about 2x to 5x faster than jiebaR.

For very long texts, splitting before tagging is usually faster than sending one huge string. In the same release benchmarks, the best results appeared around 32 to 128 chunks, while much finer splitting still helped but was no longer optimal.

When code contains multiple strings, batch controls how the per-string results are aggregated:

• "list": one single-string result per input string.

• "flatten": concatenate all results into one. The shape is decided by format: "vector"/"legacy" produce a named character vector, while "data.frame" produces a combined data frame with a doc_id column.

When batch is omitted, jiebaRS returns "vector" for single-string input and "list" for multi-string input.

Value

Tagging results in the requested format.

Examples

tagger <- worker(type = "tag")
text1 <- "\u8fd9\u662f\u4e00\u4e2a\u6d4b\u8bd5"
text2 <- "\u518d\u6765\u4e00\u6b21"
tagging(text1, tagger)
tagging(c(text1, text2), tagger)
tagging(c(text1, text2), tagger, format = "data.frame", batch = "flatten")

Tag a batch of strings

Description

Convenience wrapper around tagging() for multi-string input. When batch is not supplied, tagging_batch() always returns list output.

Usage

tagging_batch(
  texts,
  jiebar,
  ...,
  format = c("vector", "data.frame", "legacy"),
  batch = c("list", "flatten")
)

Arguments

Details

tagging_batch() is a convenience wrapper for explicit multi-string input. The returned object depends on both format and batch:

• batch = "list": returns one single-string tagging result per input string.

• batch = "flatten": concatenates all results into one. The shape is decided by format: "vector"/"legacy" produce a named character vector, while "data.frame" produces a combined data frame with a doc_id column.

In the current release benchmarks on the bundled Fortress Besieged and Dream of the Red Chamber texts, batch tagging is about 2x to 5x faster than the comparable jiebaR workflow on many-string inputs. For very long texts, the best throughput was usually reached by splitting into about 32 to 128 chunks, while much finer splitting still helped but was no longer optimal.

Value

Tagging results in the requested format.

Examples

tagger <- worker(type = "tag")
texts <- c("\u8fd9\u662f\u4e00\u4e2a\u6d4b\u8bd5", "\u518d\u6765\u4e00\u6b21")
tagging_batch(texts, tagger)
tagging_batch(texts, tagger, format = "legacy", batch = "flatten")

Tag multiple strings with an internal native worker.

Description

Internal bridge used by tagging() to tag many UTF-8 strings.

Usage

tagging_batch_worker(texts, worker)

Arguments

Value

A list where each element is a named list with term and tag vectors.

Tag text with an internal native worker.

Description

Internal bridge used by tagging() to tag a single UTF-8 string.

Usage

tagging_worker(text, worker)

Arguments

Value

A named list with term and tag vectors.

Extract TextRank keywords from text

Description

Extract TextRank-ranked keywords from a single in-memory string with a TextRank worker created by worker(). This is separate from keywords(), which uses TF-IDF weighting.

Usage

textrank(code, jiebar, ..., format = c("vector", "data.frame", "legacy"))

Arguments

Value

TextRank results in the requested format.

Extract TextRank keywords as a data frame

Description

Convenience wrapper around textrank() that always returns a data frame with term and weight columns.

Usage

textrank_df(x, jiebar)

Arguments

Value

A data frame with term and weight columns.

Extract TextRank keywords with an internal native worker.

Description

Internal bridge used by textrank() to extract TextRank keywords from a single UTF-8 string.

Usage

textrank_worker(text, worker)

Arguments

Value

A named list with keyword and weight vectors.

Initialize a jiebaRS worker

Description

This function can initialize a jiebaRS worker. See Details for more information.

Usage

worker(
  type = c("mix", "mp", "hmm", "full", "query", "tag", "keywords", "textrank"),
  stop_word = NULL,
  stop_word_file = NULL,
  hmm = TRUE,
  topn = 5L,
  min_keyword_length = 2L,
  idf = NULL,
  dict = NULL,
  user = NULL,
  symbol = FALSE,
  bylines = FALSE
)

Arguments

Details

The qmax argument is not supported. Although jiebaR documented qmax for query workers, the value was never actually passed to the underlying segmentation call. Similarly, the jieba-rs backend implements search-mode segmentation without a configurable query threshold. To avoid user confusion, jiebaRS omits the qmax argument entirely rather than retaining a no-op parameter.

jieba-rs does not expose dedicated public implementations for mp or hmm workers. jiebaRS therefore maps mp to cut(..., false) and hmm to cut(..., true). This is a compatibility approximation rather than a byte-for-byte reimplementation of jiebaR, and jiebaRS warns once per R session when either type is requested.

tag workers use jieba-rs tagging on top of the default mixed segmentation path, which is the closest public behavior to jiebaR.

stop_word and stop_word_file can be both supplied at once and then be merged together. Then they will be normalized.

In jiebaRS, hmm accepts either a logical scalar or a file path. A logical value controls whether the underlying jieba-rs segmentation/tagging pipeline may fall back to HMM for unknown terms. A character scalar is interpreted as a path to a custom HMM model file and enables HMM fallback with that model. The flag affects mix and query workers directly, tag workers through the underlying mixed tagging path, and keywords workers through TF-IDF keyword extraction. mp, hmm, and full workers ignore the runtime switch because their jieba-rs backends do not use this runtime switch.

dict and user load dictionary files at worker creation time. dict replaces the embedded main dictionary entirely; user accepts one or more file paths whose entries are appended, in the supplied order, to whatever main dictionary is in place (default or custom dict). Main dictionary entries use word [freq] [tag]; an omitted frequency defaults to 1. User dictionary entries may use word, ⁠word freq⁠, ⁠word tag⁠ (the legacy two-column jiebaR format), or ⁠word freq tag⁠. User frequencies must be positive integers; when omitted, a frequency is inferred from the current dictionary. Files must be UTF-8; a leading byte order mark (BOM) is allowed.

Value

A jieba_worker S3 object.