Package {Rhobots}

Agglomerative (hierarchical) clustering

Description

Wraps stats::hclust + stats::cutree. Like k-means, every document is assigned to a cluster (no noise label).

Usage

agglomerative_clustering(k, linkage = "ward.D2")

Arguments

Value

An agglomerative_clustering model object.

Examples

m <- agglomerative_clustering(k = 5L)

Refine topic representations with Maximal Marginal Relevance (MMR)

Description

After fitting a topic model, the top terms for each topic are selected purely by c-TF-IDF score, which can produce redundant terms (e.g. model, models, modeling). MMR re-ranks the candidate terms by jointly maximising their relevance to the topic and their diversity from already-selected terms.

Usage

apply_mmr(
  fit,
  encoder,
  diversity = 0.1,
  top_n = NULL,
  top_n_candidates = NULL,
  verbose = TRUE
)

Arguments

Details

The algorithm mirrors Python BERTopic's MaximalMarginalRelevance representation model:

1. For each topic, take the top top_n_candidates terms from c-TF-IDF as the candidate pool.

2. Embed every unique candidate term and a topic reference string (the candidates joined into one string) using encoder.

3. Greedily select top_n terms by

   MMR_i = (1 - \lambda)\,\text{sim}(w_i, \text{topic}) - \lambda\,\max_{s \in S}\text{sim}(w_i, s)

   where S is the set of already-selected terms and \lambda = diversity.

4. Original c-TF-IDF scores are preserved for the selected terms; only the selection and ranking change.

Note: MMR operates on the existing topic_terms in the fit object. If you subsequently call reduce_topics, merge_topics, or reduce_outliers, topic terms are recomputed from c-TF-IDF and MMR needs to be re-applied.

Value

An updated bertopic_fit with topic_terms and topic_labels replaced by the MMR-selected representation.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  fit <- apply_mmr(fit, encoder = enc, diversity = 0.3)
  get_topic(fit, 0L)

## End(Not run)

Build a sparse document-term matrix from a character vector

Description

Tokenizes each document (lowercase, split on non-alphanumeric), removes stopwords, optionally generates n-grams, then applies document-frequency filtering. Returns a sparse matrix suitable for the c-TF-IDF step.

Usage

build_dtm(
  docs,
  min_df = 2,
  max_df_frac = 0.95,
  stopwords = character(),
  ngram_range = c(1L, 1L)
)

Arguments

Value

A sparse dgCMatrix of dimensions length(docs) x vocab size, with colnames set to the vocabulary.

Examples

docs <- c("the cat sat on the mat", "the dog chased the cat",
          "a cat and a dog", "the mat is on the floor")
build_dtm(docs, min_df = 1L)

Class-based TF-IDF (c-TF-IDF) for cluster-level topic terms

Description

Treats each cluster as one big document and ranks terms by class TF multiplied by a class-based inverse document frequency. This is the BERTopic-flavoured TF-IDF that produces interpretable topic descriptors.

Usage

c_tf_idf(dtm, cluster_ids, top_n = 10, reduce_frequent_words = FALSE)

Arguments

Value

A data frame with columns topic, rank, term, score.

Examples

docs <- c("the cat sat on mat", "a dog chased cat",
          "machine learning models", "deep learning neural nets")
dtm  <- build_dtm(docs, min_df = 1L)
c_tf_idf(dtm, cluster_ids = c(0L, 0L, 1L, 1L), top_n = 3L)

Classify or label texts with a fine-tuned BERT-family model

Description

S3 generic that dispatches on the classifier class returned by load_hf_classifier().

Usage

classify_texts(classifier, texts, ...)

## S3 method for class 'hf_classifier'
classify_texts(
  classifier,
  texts,
  batch_size = 32L,
  max_length = 512L,
  verbose = FALSE,
  ...
)

## Default S3 method:
classify_texts(classifier, texts, ...)

Arguments

Details

Sequence classification (sentiment, topic, ...): returns a data.frame with columns text, label, score, plus one probability column per label. For problem_type = "regression" the label columns contain raw numeric scores. For problem_type = "multi_label_classification" each label column contains a sigmoid probability and there is no single label or score column.

Token classification / NER: returns a named list of data.frames, one per input text, each with columns token, label, score. Special tokens (⁠[CLS]⁠, ⁠[SEP]⁠, padding) are automatically excluded.

Value

For sequence tasks: a data.frame with nrow(texts) rows. For NER: a list of data.frames, one per input text.

Examples

## Not run: 
  clf  <- load_hf_classifier("cardiffnlp/twitter-xlm-roberta-base-sentiment")
  res  <- classify_texts(clf, c("I love this!", "Terrible experience."))
  res$label   # c("positive", "negative")
  res$score   # highest class probability

  # VAD regression
  clf <- load_hf_classifier("RobroKools/vad-bert")
  classify_texts(clf, c("I am ecstatic!"))
  # returns: text | valence | arousal | dominance

  # NER
  clf <- load_hf_classifier("dslim/bert-base-NER")
  result <- classify_texts(clf, c("Marie Curie was born in Warsaw."))
  result[[1]]   # token | label | score

## End(Not run)

CLS-token pooling: extract the CLS hidden state as the sentence vector

Description

Returns the first token's hidden state ⁠(B, H)⁠ from a ⁠(B, L, H)⁠ tensor. Used for models whose ⁠1_Pooling/config.json⁠ sets pooling_mode_cls_token = true (e.g. some BGE and GTE variants).

Usage

cls_pool(hidden)

Arguments

Value

A 2-D torch_tensor of shape ⁠(batch, hidden)⁠.

Examples

## Not run: 
  enc    <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  tokens <- enc$tokenizer$encode_batch(c("hello world"))
  ids    <- torch::torch_tensor(matrix(tokens[[1L]]$ids, nrow = 1L),
                                 dtype = torch::torch_long())
  mask   <- torch::torch_tensor(matrix(tokens[[1L]]$attention_mask, nrow = 1L),
                                 dtype = torch::torch_long())
  hidden <- enc$model(ids, mask)
  cls_pool(hidden)

## End(Not run)

Fit a clustering model and return cluster labels

Description

Fit a clustering model and return cluster labels

Usage

cluster_docs(model, X, seed = 42L)

Arguments

Value

A list with $labels (integer vector; -1 = noise, 0, 1, 2, ... = cluster IDs) and $model (the fitted model).

Examples

## Not run: 
  m   <- hdbscan_clustering(min_pts = 3L)
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  emb <- embed_texts(enc, c("cats and dogs", "machine learning",
                             "pets and animals", "neural networks"))
  res <- cluster_docs(m, emb)
  res$labels

## End(Not run)

Compare topic prevalence across groups

Description

Given a grouping variable (e.g. publication year, country, experimental condition) of the same length as the corpus, computes which topics are over- or under-represented in each group relative to a null model of independence.

Usage

compare_topics(
  fit,
  groups,
  method = c("chi2", "log_ratio"),
  min_count = 5L,
  verbose = TRUE
)

Arguments

Details

Two statistics are available:

"chi2"

Signed chi-square contribution: sign(O-E) \cdot \sqrt{(O-E)^2/E}. Positive = over-represented, negative = under-represented. A global chi-square test is also reported.

"log_ratio"

Laplace-smoothed log_2 ratio of observed to expected proportion. Values above 1 mean the topic is twice as prevalent in that group as expected; below -1 means half as prevalent.

Value

A list of class compare_topics_result with elements:

result

Tidy data frame with columns Topic, Name, Group, Observed, Expected, Stat.

table

Raw contingency table (topics x groups).

global_statistic, global_pvalue

Global chi-square statistic and p-value (NA when method = "log_ratio").

See Also

visualize_comparison

Examples

## Not run: 
  enc    <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit    <- fit_bertopic(docs = abstracts, encoder = enc)
  groups <- sample(c("A", "B"), length(abstracts), replace = TRUE)
  comp   <- compare_topics(fit, groups)
  visualize_comparison(comp)

## End(Not run)

Construct a C-value representation model

Description

Use as the representation_model argument in fit_bertopic to select the n-gram vocabulary using the C-value algorithm (Frantzi et al. 2000) rather than plain document-frequency filtering. Unigrams are always kept alongside the C-value-selected multi-word terms.

Usage

cvalue_representation(max_n = 4L, threshold = 0, min_freq = 2L)

Arguments

Value

An object of class cvalue_representation.

See Also

cvalue_terms, fit_bertopic, pos_representation

Examples

m <- cvalue_representation(max_n = 3L, threshold = 0.5)

Compute C-value scores for candidate multi-word terms

Description

Implements the C-value method of Frantzi, Ananiadou & Mima (2000). For each candidate n-gram t (n >= 2):

Usage

cvalue_terms(
  docs,
  max_n = 4L,
  min_freq = 2L,
  threshold = 0,
  stopwords = character(0L)
)

Arguments

Details

C\text{-value}(t) = \log_2|t| \times \begin{cases} f(t) & \text{if } P(t) = \emptyset \\ f(t) - \dfrac{1}{|P(t)|} \displaystyle\sum_{a \in P(t)} f(a) & \text{otherwise} \end{cases}

where |t| is word count, f(t) is corpus frequency, and P(t) is the set of longer candidate terms that contain t as a contiguous sub-sequence.

Value

A data frame with columns term (underscore-separated tokens), freq, n_words, and cvalue, sorted by descending C-value. Terms with cvalue < threshold are excluded.

References

Frantzi, K., Ananiadou, S., & Mima, H. (2000). Automatic recognition of multi-word terms: the C-value/NC-value method. International Journal on Digital Libraries, 3(2), 115 - 130.

See Also

cvalue_representation, fit_bertopic

Examples

docs <- c("sea level rise and climate change", "sea level is rising",
          "climate change impacts sea level", "Arctic ice melt sea level")
cvalue_terms(docs, max_n = 3L, min_freq = 2L)

Project new data using a fitted dimensionality-reduction model

Description

Project new data using a fitted dimensionality-reduction model

Usage

dim_project(model, X)

Arguments

Value

A numeric matrix with the same number of columns as the training embedding.

Examples

## Not run: 
  m    <- umap_reduction(n_neighbors = 5L, n_components = 2L)
  enc  <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  emb  <- embed_texts(enc, c("first doc", "second doc", "third doc"))
  res  <- dim_reduce(m, emb)
  new_emb <- embed_texts(enc, c("new document"))
  dim_project(res$model, new_emb)

## End(Not run)

Fit a dimensionality-reduction model and return the reduced matrix

Description

Fit a dimensionality-reduction model and return the reduced matrix

Usage

dim_reduce(model, X, seed = 42L, verbose = FALSE)

Arguments

Value

A list with $embedding (reduced matrix) and $model (the fitted model, for use with dim_project).

Examples

## Not run: 
  m   <- umap_reduction(n_neighbors = 5L, n_components = 2L)
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  emb <- embed_texts(enc, c("first doc", "second doc", "third doc"))
  res <- dim_reduce(m, emb)

## End(Not run)

Embed a vector of texts to a numeric matrix

Description

S3 generic dispatching on the encoder class. For local BERT-family models (class bert_encoder) see the Details section. For API-backed models (class api_embedder) the function calls the provider's REST endpoint and parameters max_length, device, chunk_strategy, and chunk_overlap are ignored.

Usage

## S3 method for class 'api_embedder'
embed_texts(
  encoder,
  texts,
  batch_size = NULL,
  normalize = TRUE,
  prefix = NULL,
  verbose = interactive(),
  ...
)

embed_texts(encoder, texts, ...)

## S3 method for class 'bert_encoder'
embed_texts(
  encoder,
  texts,
  batch_size = 32L,
  max_length = 256L,
  normalize = TRUE,
  device = "cpu",
  prefix = NULL,
  chunk_strategy = c("truncate", "mean", "first"),
  chunk_overlap = 0L,
  verbose = interactive(),
  ...
)

## Default S3 method:
embed_texts(encoder, texts, ...)

Arguments

Details

Tokenizes, runs the encoder forward pass, pools over tokens (mean or CLS, auto-detected from the encoder's pooling configuration), and optionally L2-normalizes the result. Batches inputs for memory efficiency.

An instruction prefix can be prepended to every text before tokenization. This is required for best performance with BGE and E5 models:

• BGE (⁠BAAI/bge-*⁠): prefix = "Represent this sentence: "

• E5 (⁠intfloat/e5-*⁠): prefix = "passage: "

If the encoder was loaded with a prefix via load_hf_bert()'s prefix argument, that value is used automatically and need not be repeated here. Passing prefix explicitly always takes precedence.

Long-document chunking (chunk_strategy): BERT-family models have a fixed maximum sequence length (usually 512 tokens) and silently truncate longer inputs. Setting chunk_strategy = "mean" (or "first") instead splits each over-length text into overlapping windows of max_length tokens, embeds each window, and aggregates:

• "truncate" (default): truncate at max_length, fast, no overhead.

• "mean": embed all windows, average their vectors (then normalize if normalize = TRUE). Best quality for long documents.

• "first": embed only the first window. Good when the lead of a document (abstract, summary) is the most informative part.

Use chunk_overlap to set the number of overlapping tokens between consecutive windows (default 0).

Value

A numeric matrix with length(texts) rows and hidden_size cols.

Examples

## Not run: 
  # General model  --  no prefix needed
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  emb <- embed_texts(enc, c("First sentence.", "Second sentence."))

  # BGE model  --  set prefix at load time (applied automatically)
  enc <- load_hf_bert("BAAI/bge-base-en-v1.5",
                       prefix = "Represent this sentence: ")
  emb <- embed_texts(enc, docs)

  # E5 model  --  passage prefix for documents, query prefix for queries
  enc <- load_hf_bert("intfloat/e5-base-v2", prefix = "passage: ")
  doc_emb   <- embed_texts(enc, docs)
  query_emb <- embed_texts(enc, queries, prefix = "query: ")

  # Long documents: chunk and mean-pool windows
  enc <- load_hf_bert("sentence-transformers/all-mpnet-base-v2")
  emb <- embed_texts(enc, long_papers, chunk_strategy = "mean",
                      chunk_overlap = 32L)

  # API-based (no torch required)
  enc <- load_openai_embedder()
  emb <- embed_texts(enc, docs)

## End(Not run)

Compute or load document embeddings from a cache file

Description

On the first call (or when overwrite = TRUE) the encoder is used to compute embeddings and the result is written to cache_file. On subsequent calls the cached matrix is read directly, making it free to experiment with different dimensionality-reduction or clustering settings without re-running the expensive forward passes.

Usage

embed_texts_cached(
  encoder = NULL,
  texts,
  cache_file = NULL,
  overwrite = FALSE,
  batch_size = 32L,
  max_length = 256L,
  normalize = TRUE,
  device = "cpu",
  prefix = NULL,
  chunk_strategy = c("truncate", "mean", "first"),
  chunk_overlap = 0L,
  verbose = interactive()
)

Arguments

Value

A numeric matrix with length(texts) rows.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  docs <- c("First document.", "Second document.", "Third document.")
  emb  <- embed_texts_cached(enc, docs, cache_file = tempfile(fileext = ".rds"))

## End(Not run)

Find topics most similar to a search term

Description

Ranks non-noise topics by the c-TF-IDF score of search_term. If the term is absent from the vocabulary entirely, falls back to substring matching across each topic's top terms.

Usage

find_topics(fit, search_term, top_n = 5L)

Arguments

Value

A data frame with columns Topic, Name, Score, sorted by descending score.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  find_topics(fit, "neural network")

## End(Not run)

Fit a BERTopic-style topic model

Description

Runs the four-stage BERTopic pipeline:

1. Embed documents with the supplied encoder.

2. Reduce embedding dimensionality with UMAP.

3. Cluster the reduced space with HDBSCAN.

4. Extract per-topic terms with c-TF-IDF.

Usage

fit_bertopic(
  encoder = NULL,
  docs,
  embeddings = NULL,
  dim_reduction_model = NULL,
  cluster_model = NULL,
  representation_model = NULL,
  umap_n_neighbors = 15,
  umap_n_components = 5,
  umap_min_dist = 0,
  umap_metric = "cosine",
  hdbscan_min_pts = 10,
  hdbscan_method = c("eom", "leaf"),
  top_n_terms = 10,
  language = "english",
  stopwords = NULL,
  extra_stopwords = NULL,
  ngram_range = c(1L, 1L),
  reduce_frequent_words = FALSE,
  seed = 42,
  verbose = TRUE
)

Arguments

Details

Also computes a separate 2-D UMAP projection of the same embeddings, for visualization.

This implementation will be extended over time to mirror more of the Python BERTopic package's capabilities (custom vectorizers, topic reduction, representation models, dynamic topic modeling).

Value

A list (class bertopic_fit) with elements: embeddings, reduced, layout2d, clusters, topic_terms, hdbscan, docs.

Examples

## Not run: 
  enc <- load_hf_bert("pritamdeka/S-Scibert-snli-multinli-stsb")
  fit <- fit_bertopic(enc, docs = my_abstracts)
  print_topics(fit)

## End(Not run)

Fit independent BERTopic models per time period and align topics

Description

All documents are embedded once using the shared encoder, then one BERTopic model is fitted per time period using the appropriate slice of embeddings. Because every model lives in the same embedding space, cosine similarity between topic centroids across periods is a meaningful measure of topic continuity.

Usage

fit_topics_over_time(
  encoder,
  docs,
  timestamps,
  min_similarity = 0.7,
  bertopic_params = list(),
  verbose = TRUE
)

Arguments

Details

Adjacent periods are compared by computing the full cosine-similarity matrix between their topic centroids. Any pair whose similarity exceeds min_similarity becomes a directed link in the transition graph. From that graph each topic is labelled:

emerges

No incoming link from the previous period.

disappears

No outgoing link to the next period.

continues

Exactly one incoming and one outgoing link.

splits

One incoming link, multiple outgoing links.

merges

Multiple incoming links, one outgoing link.

isolated

No links in either direction.

Value

An object of class bertopic_flow containing:

periods

Character vector of period labels in order.

fits

Named list of bertopic_fit objects.

transitions

Data frame of inter-period topic links: period_from, topic_from, period_to, topic_to, similarity.

topic_info

Data frame with one row per (period, topic): period, topic, label, count, status.

Examples

## Not run: 
  enc    <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  period <- rep(c("2010s", "2020s"), each = 50L)
  flow   <- fit_topics_over_time(enc, docs = abstracts,
                                  period_var = period)
  visualize_topic_flow(flow)

## End(Not run)

Get document-level topic assignments as a data frame

Description

Get document-level topic assignments as a data frame

Usage

get_document_info(fit, metadata = NULL)

Arguments

Value

A data frame with one row per document and columns Document, Topic, Name, Top_words.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  get_document_info(fit)

## End(Not run)

Get representative documents for one or all topics

Description

Representative documents are the three training documents whose embeddings lie closest (cosine similarity) to the topic centroid.

Usage

get_representative_docs(fit, topic = NULL)

Arguments

Value

A character vector (single topic) or a named list of character vectors (all topics).

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  get_representative_docs(fit, topic = 0L)

## End(Not run)

Return the built-in stopword list for a language

Description

Currently only "english" is supported; returns an empty character vector for any other value.

Usage

get_stopwords(language = "english")

Arguments

Value

A character vector of stopwords.

Examples

get_stopwords("english")

Get term-score representation for a single topic

Description

Get term-score representation for a single topic

Usage

get_topic(fit, topic)

Arguments

Value

A data frame with columns term and score, or NULL if the topic ID is not found.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  get_topic(fit, topic = 0L)

## End(Not run)

Get topic-level metadata as a data frame

Description

Get topic-level metadata as a data frame

Usage

get_topic_info(fit, topic = NULL)

Arguments

Value

A data frame with columns:

Topic

Integer topic ID (-1 = noise/unassigned).

Count

Number of documents assigned to this topic.

Name

Auto-generated label (e.g. "0_model_data_...").

Representation

Top-5 terms, comma-separated.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  get_topic_info(fit)

## End(Not run)

Get all topic-term representations

Description

Get all topic-term representations

Usage

get_topics(fit, top_n = NULL)

Arguments

Value

A named list; each element is a data frame with columns term and score sorted by descending score. Names are topic IDs (character strings, including "-1" for noise).

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  get_topics(fit, top_n = 5L)

## End(Not run)

Guided topic modeling with user-supplied seed words

Description

Extends fit_bertopic with a seed_topic_list argument. For each seed topic the user supplies a character vector of representative words; these are concatenated into virtual "anchor" documents, embedded, and added to the corpus before dimensionality reduction and clustering. The anchors bias the latent space so that documents semantically close to the seed words cluster together. Seed documents are removed from the fit before the object is returned.

Usage

guided_fit_bertopic(
  docs,
  seed_topic_list,
  embeddings = NULL,
  encoder = NULL,
  n_anchor_weight = 3L,
  ...,
  verbose = TRUE
)

Arguments

Value

A bertopic_fit object. The attribute seed_map records which cluster each seed topic was assigned to.

See Also

fit_bertopic, zero_shot_topics

Examples

## Not run: 
  enc   <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  seeds <- list(
    "Climate" = c("carbon", "emissions", "greenhouse", "warming"),
    "Economy" = c("trade", "wages", "labour", "inflation")
  )
  fit <- guided_fit_bertopic(abstracts, seed_topic_list = seeds, encoder = enc)

## End(Not run)

HDBSCAN via Ball-tree dual-tree Borůvka MST

Description

Default internal HDBSCAN implementation (called when knn = "balltree"). Builds a Ball-tree from the data matrix and runs dual-tree Borůvka. Ball bounding spheres prune more effectively than axis-aligned boxes in >=3-D, keeping Borůvka rounds close to O(n log n) even on data without strong cluster separation. kNN results from the core-distance pass are reused as a Borůvka warm-up, so no extra tree traversal is needed.

Usage

hdbscan_balltree_cpp(X, min_pts, allow_single_cluster = FALSE)

Arguments

Value

Named list: labels (IntegerVector, 0 = noise), n_mst_edges (int, always n-1 when data is connected).

HDBSCAN via Boruvka MST on mutual-reachability kNN graph

Description

Internal function called by cluster_docs.hdbscan_clustering(). Returns a list with labels (integer vector, 0 = noise) and n_mst_edges (number of MST edges built). When n_mst_edges < n-1 the kNN graph was disconnected; the R wrapper retries with larger k.

Usage

hdbscan_boruvka_cpp(knn_idx, knn_dist, min_pts, allow_single_cluster = FALSE)

Arguments

Value

Named list: labels (IntegerVector), n_mst_edges (int).

HDBSCAN clustering

Description

HDBSCAN density clustering

Usage

hdbscan_clustering(
  min_pts = 10L,
  method = c("eom", "leaf"),
  knn = c("balltree", "kdtree", "adaptive", "fixed"),
  allow_single_cluster = FALSE
)

Arguments

Details

Default clustering model for fit_bertopic. Uses a Boruvka minimum spanning tree on the mutual-reachability kNN graph (Rcpp) instead of the naive Prim's algorithm in dbscan::hdbscan(), which avoids the O(n^2) memory cost that causes OOM at ~30K+ documents. Memory is O(n x k) throughout, making it practical at 100K+ documents.

Value

An hdbscan_clustering model object.

Examples

m <- hdbscan_clustering(min_pts = 5L)
m_kd <- hdbscan_clustering(min_pts = 5L, knn = "kdtree")

HDBSCAN via Boruvka MST with KD-tree queries (no fixed k)

Description

Internal function called by cluster_docs.hdbscan_clustering() when knn = "kdtree". Unlike hdbscan_boruvka_cpp(), this function takes the raw data matrix X and builds its own KD-tree (nanoflann), querying it on-demand during each Boruvka round. No k parameter is exposed: the search radius grows automatically until the MST is fully connected. This replicates the behaviour of Python hdbscan's boruvka_kdtree algorithm.

Usage

hdbscan_kdtree_cpp(X, min_pts, allow_single_cluster = FALSE)

Arguments

Value

Named list: labels (IntegerVector, 0 = noise), n_mst_edges (int, always n-1 when data is connected).

Build a hierarchical topic tree from a fitted topic model

Description

Performs agglomerative clustering on the L2-normalised topic centroids stored in fit$topic_centroids using cosine distance. At each merge node the function pools the c-TF-IDF scores of all constituent topics and records the top terms, so every level of the hierarchy is labelled.

Usage

hierarchical_topics(fit, method = "ward.D2", top_n_terms = 10L)

Arguments

Details

The result can be visualised with visualize_hierarchy().

Value

A list of class hierarchical_topics with elements:

hclust

the hclust object (for use with base R dendrogram functions if desired)

merge_df

a data frame with one row per merge, recording parent_id, child_left, child_right, distance (cosine), topics (comma-separated topic IDs), and terms

topic_ids

integer vector of leaf topic IDs in original order

method

the linkage method used

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(enc, docs = abstracts)
  h   <- hierarchical_topics(fit)
  print(h)
  visualize_hierarchy(h, fit = fit)

## End(Not run)

K-means clustering

Description

Wraps stats::kmeans. Unlike HDBSCAN, k-means assigns every document to a cluster (no noise label -1) and requires specifying the number of clusters k in advance.

Usage

kmeans_clustering(k, nstart = 10L, iter.max = 300L)

Arguments

Value

A kmeans_clustering model object.

Examples

m <- kmeans_clustering(k = 5L)

Label topics using a large language model

Description

For each non-noise topic, builds a prompt containing the top c-TF-IDF terms and up to n_representative_docs representative documents, sends it to an LLM API, and stores the returned label in fit$topic_labels.

Usage

label_topics_llm(
  fit,
  provider = c("anthropic", "openai", "ollama"),
  api_key = NULL,
  model = NULL,
  top_n_terms = 10L,
  n_representative_docs = 3L,
  custom_prompt = NULL,
  verbose = TRUE
)

Arguments

Details

Three providers are supported:

"anthropic"

Anthropic Claude via the Messages API. Requires api_key or the ANTHROPIC_API_KEY environment variable.

"openai"

OpenAI GPT via the Chat Completions API. Requires api_key or OPENAI_API_KEY.

"ollama"

Local Ollama server (OpenAI-compatible endpoint at http://localhost:11434/v1). No API key required. Start the server with ollama serve and pull a model with ollama pull llama3.2 before use.

Requires the httr2 package.

Value

The input fit with updated $topic_labels.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  fit <- label_topics_llm(fit, provider = "anthropic",
                          api_key = Sys.getenv("ANTHROPIC_API_KEY"),
                          model = "claude-haiku-4-5-20251001")
  get_topic_info(fit)

## End(Not run)

Load BERT weights from a checkpoint into a constructed model

Description

Reads weights from a safetensors or PyTorch pickle file, normalizes the parameter names to match the R module structure, filters out task-head keys we don't need (⁠pooler.*⁠, ⁠cls.*⁠, ⁠lm_head.*⁠, ...), and applies the result via model$load_state_dict().

Usage

load_bert_weights(model, weights_path, strict = FALSE)

Arguments

Details

Most users don't call this directly – it's invoked by load_hf_bert(). Exposed for users who construct a model manually or want to load alternative weight files.

Value

Invisibly, the named list of loaded weights.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  # load_bert_weights() is called internally by load_hf_bert() and
  # load_specter2(); advanced users can call it directly when injecting
  # custom weights into an existing model object.

## End(Not run)

Load a previously saved BERTopic model

Description

Reads a model directory written by save_bertopic and reconstructs the bertopic_fit object.

Usage

load_bertopic(path)

Arguments

Value

A bertopic_fit object.

See Also

save_bertopic

Examples

## Not run: 
  enc  <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit  <- fit_bertopic(docs = abstracts, encoder = enc)
  path <- tempdir()
  save_bertopic(fit, path)
  fit2 <- load_bertopic(path)

## End(Not run)

Load a Cohere embedding model

Description

Returns an api_embedder that calls the Cohere Embed API. Requires the httr2 package and a valid Cohere API key.

Usage

load_cohere_embedder(
  model = "embed-english-v3.0",
  api_key = Sys.getenv("COHERE_API_KEY"),
  input_type = "search_document"
)

Arguments

Details

The returned object is compatible with embed_texts(), embed_texts_cached(), and fit_bertopic().

Value

An api_embedder object.

Examples

## Not run: 
  enc <- load_cohere_embedder()   # uses COHERE_API_KEY env var
  emb <- embed_texts(enc, docs)

## End(Not run)

Load an embedding matrix from disk

Description

Load an embedding matrix from disk

Usage

load_embeddings(path)

Arguments

Value

A numeric matrix.

Examples

## Not run: 
  enc  <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  emb  <- embed_texts(enc, c("doc one", "doc two"))
  path <- tempfile(fileext = ".rds")
  save_embeddings(emb, path)
  load_embeddings(path)

## End(Not run)

Load a BERT-family model from HuggingFace for use in R

Description

Downloads a model's config, weights, and tokenizer from the HuggingFace Hub and returns an "encoder" object that can be passed to embed_texts() or fit_bertopic(). Works for any model with a standard BERT architecture: vanilla BERT, MiniLM, MPNet, SciBERT, BioBERT, ClinicalBERT, BERT-for-Patents, FinBERT, LegalBERT, the sentence-transformers built on top of these, and so on.

Usage

load_hf_bert(repo_id, weights_path = NULL, prefix = "")

Arguments

Details

The function tries to use modern formats when available and falls back transparently when they aren't:

• Weights: model.safetensors is preferred (device-agnostic, safe, fast). If absent, pytorch_model.bin is read via R-torch's pickle loader. If the model has only pytorch_model.bin and that file was saved on a CUDA device, the load will fail – convert the upstream model to safetensors first (see HuggingFace's safetensors/convert Space) and pass the result via weights_path.

• Tokenizer: tokenizer.json (HuggingFace fast tokenizer via the tok package) is preferred. If absent, falls back to make_wordpiece_tokenizer() reading vocab.txt (requires the wordpiece package).

Value

A list with three elements:

model

the loaded bert_model nn_module

tokenizer

a tokenizer object exposing encode_batch()

config

the architecture config as a list

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  emb <- embed_texts(enc, c("Hello world.", "Another sentence."))

  # BGE model  --  prefix at load time, applied automatically on every embed call
  enc <- load_hf_bert("BAAI/bge-base-en-v1.5",
                       prefix = "Represent this sentence: ")

  # E5 model  --  passage prefix for documents
  enc <- load_hf_bert("intfloat/e5-base-v2", prefix = "passage: ")
  # Override per-call for query-side embedding:
  query_emb <- embed_texts(enc, queries, prefix = "query: ")

  # With a custom weights file (e.g. safetensors from an unmerged PR)
  enc <- load_hf_bert("pritamdeka/S-Scibert-snli-multinli-stsb",
                      weights_path = "/path/to/local/model.safetensors")

## End(Not run)

Load a fine-tuned BERT-family classifier from HuggingFace

Description

Downloads config.json, model weights, and a tokenizer from the HuggingFace Hub and returns an hf_classifier object. Supports sequence classification (sentiment, topic, intent), multi-label classification, regression (e.g. VAD emotion scores), and token classification (NER, POS tagging).

Usage

load_hf_classifier(repo_id, weights_path = NULL, prefix = "")

Arguments

Details

The task type and output format are auto-detected from config.json:

• architectures containing "ForTokenClassification" -> NER mode

• problem_type = "regression" -> return raw numeric scores

• problem_type = "multi_label_classification" -> sigmoid per label

• otherwise -> softmax single-label classification

Supported backbone architectures: the same as load_hf_bert() – BERT, RoBERTa, XLM-RoBERTa, CamemBERT, DistilBERT (backbone only), MPNet.

Value

An hf_classifier list with elements model, tokenizer, config, id2label, problem_type, task, num_labels, repo_id, prefix.

Examples

## Not run: 
  # Sentiment analysis (single-label, 3 classes)
  clf <- load_hf_classifier("cardiffnlp/twitter-xlm-roberta-base-sentiment")
  classify_texts(clf, c("I love this!", "Terrible experience."))

  # VAD regression (valence, arousal, dominance)
  clf <- load_hf_classifier("RobroKools/vad-bert")
  classify_texts(clf, c("I am ecstatic!", "Feeling nervous about the exam."))

  # Named entity recognition
  clf <- load_hf_classifier("dslim/bert-base-NER")
  classify_texts(clf, c("Albert Einstein was born in Ulm, Germany."))

## End(Not run)

Load an OpenAI embedding model

Description

Returns an api_embedder that calls the OpenAI Embeddings API. Requires the httr2 package and a valid OpenAI API key.

Usage

load_openai_embedder(
  model = "text-embedding-3-small",
  api_key = Sys.getenv("OPENAI_API_KEY"),
  dimensions = NULL,
  base_url = "https://api.openai.com/v1"
)

Arguments

Details

The returned object is compatible with embed_texts(), embed_texts_cached(), and fit_bertopic().

Value

An api_embedder object.

Examples

## Not run: 
  enc <- load_openai_embedder()   # uses OPENAI_API_KEY env var
  emb <- embed_texts(enc, c("Transformers in R.", "Topic modelling."))
  fit <- fit_bertopic(enc, docs = abstracts)

## End(Not run)

Load a SPECTER2 model with a task-specific adapter

Description

SPECTER2 doi:10.48550/arXiv.2211.13308 extends the original SPECTER model with task-specific Pfeiffer adapters trained on millions of citation pairs. This function loads the base encoder from allenai/specter2_base and injects the chosen adapter into each transformer layer, matching the behaviour of the Python adapters library.

Usage

load_specter2(
  adapter = "NetworkIsLife/specter2",
  base = "NetworkIsLife/specter2_base",
  adapter_name = "[PRX]"
)

Arguments

Details

Available adapters on the HuggingFace Hub:

"allenai/specter2"

Proximity / similarity – recommended for document retrieval and topic modeling (default).

"allenai/specter2_adhoc_query"

Query-side adapter for asymmetric retrieval (query vs. document).

"allenai/specter2_classification"

Trained for paper classification tasks.

The returned object is a standard bert_encoder compatible with embed_texts(), embed_texts_cached(), and fit_bertopic().

Value

A bert_encoder object (same class as load_hf_bert()) with Pfeiffer adapter modules injected into every transformer layer. The list also carries ⁠$adapter_repo⁠ recording which adapter was loaded.

Examples

## Not run: 
  enc <- load_specter2()   # proximity adapter  --  best for topic modeling
  emb <- embed_texts(enc, c("Graph neural networks for drug discovery.",
                             "Climate tipping points and carbon budgets."))

## End(Not run)

Load a stopword list from a character vector, data frame, or file

Description

Accepts three input types, making it easy to manage domain-specific stopwords from any source:

Character vector

Returned as-is (after deduplication and trimming).

Data frame

Words are taken from column (or the first column when column is NULL).

File path

Supports two file types:

• Plain text (.txt or no extension): one word per line, blank lines are ignored.

• Tabular (.csv or .tsv): read as a data frame and words extracted from column (or the first column).

Usage

load_stopwords(source, column = NULL)

Arguments

Details

The result is typically combined with the built-in list before passing to fit_bertopic:

  domain_words <- load_stopwords("domain_stop.txt")
  fit <- fit_bertopic(enc, docs,
                      extra_stopwords = domain_words)

Value

A deduplicated character vector of stopwords (lowercased and whitespace-trimmed).

Examples

load_stopwords(c("foo", "bar", "baz"))
load_stopwords(data.frame(word = c("foo", "bar")), column = "word")

Create a WordPiece tokenizer for BERT models that lack tokenizer.json

Description

Older BERT-family models (SciBERT, BioBERT, the original BERT-base, etc.) ship a vocab.txt file but not the HuggingFace fast-tokenizer format. This function builds a tokenizer object exposing the same methods as tok::tokenizer$from_pretrained() (encode_batch, enable_padding, enable_truncation) so it can be used interchangeably with embed_texts().

Usage

make_wordpiece_tokenizer(vocab_path, do_lower_case = NULL, max_length = 512L)

Arguments

Details

Uses the CRAN package wordpiece for the WordPiece algorithm itself.

Value

A list with methods encode_batch(texts), enable_padding(), and enable_truncation(max_length).

Examples

## Not run: 
  vocab <- hfhub::hub_download("allenai/scibert_scivocab_cased", "vocab.txt")
  tk <- make_wordpiece_tokenizer(vocab)
  tk$encode_batch(c("This is a sentence.", "Another one."))

## End(Not run)

Mean-pool token-level hidden states into a sentence vector

Description

Takes the encoder's final hidden states ⁠(B, L, H)⁠ and the attention mask ⁠(B, L)⁠, masks out padding positions, and averages along the sequence dimension to produce ⁠(B, H)⁠ sentence vectors.

Usage

mean_pool(hidden, attention_mask)

Arguments

Value

A 2-D torch_tensor of shape ⁠(batch, hidden)⁠.

Examples

## Not run: 
  enc    <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  tokens <- enc$tokenizer$encode_batch(c("hello world"))
  ids    <- torch::torch_tensor(matrix(tokens[[1L]]$ids, nrow = 1L),
                                 dtype = torch::torch_long())
  mask   <- torch::torch_tensor(matrix(tokens[[1L]]$attention_mask, nrow = 1L),
                                 dtype = torch::torch_long())
  hidden <- enc$model(ids, mask)
  mean_pool(hidden, mask)

## End(Not run)

Manually merge a set of topics into one

Description

All topics in topics_to_merge are combined into the one with the smallest ID. All derived state (c-TF-IDF terms, labels, centroids, representative docs) is recomputed from the merged assignments.

Usage

merge_topics(fit, topics_to_merge)

Arguments

Value

An updated bertopic_fit.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  fit <- merge_topics(fit, topics_to_merge = c(0L, 1L))

## End(Not run)

Skip dimensionality reduction (identity pass-through)

Description

Passes the raw embeddings directly to the clustering step. Useful when the embeddings are already low-dimensional or when you want to cluster in the original space.

Usage

no_reduction()

Value

A no_reduction model object.

Examples

m <- no_reduction()

PCA dimensionality reduction

Description

Wraps stats::prcomp. Unlike UMAP, PCA is deterministic and fast, but often produces lower-quality cluster separation for text embeddings.

Usage

pca_reduction(n_components = 5L, scale. = FALSE)

Arguments

Value

A pca_reduction model object.

Examples

m <- pca_reduction(n_components = 3L)

Build a POS-filtered document-term matrix

Description

Annotates each document in docs with a udpipe language model, keeps only tokens whose UPOS tag is in pos (and/or phrases matching patterns), then builds a sparse document-term matrix from the surviving tokens. The result is compatible with c_tf_idf.

Usage

pos_dtm(
  docs,
  pos = c("NOUN", "PROPN"),
  patterns = NULL,
  lemmatize = TRUE,
  language = "english",
  model_dir = NULL,
  min_df = 2L,
  max_df_frac = 0.95,
  extra_stopwords = character(0L),
  verbose = TRUE
)

Arguments

Value

A sparse dgCMatrix: documents x POS-filtered vocabulary.

See Also

pos_representation, build_dtm

Examples

## Not run: 
  docs <- c("The neural network learns features.",
            "Gradient descent optimizes weights.",
            "Transformers use attention mechanisms.")
  pos_dtm(docs, pos = c("NOUN", "VERB"))

## End(Not run)

Construct a POS-based representation model

Description

Use as the representation_model argument in fit_bertopic to restrict the c-TF-IDF vocabulary to tokens whose Universal POS (UPOS) tag is in pos, or to phrases that match specific POS-tag sequences.

Usage

pos_representation(
  pos = c("NOUN", "PROPN"),
  patterns = NULL,
  lemmatize = TRUE,
  language = "english",
  model_dir = NULL,
  min_df = 2L,
  max_df_frac = 0.95
)

Arguments

Details

Requires the udpipe package (install.packages("udpipe")). A language model (~15 MB) is downloaded from the Universal Dependencies collection on first use and cached in model_dir.

Common UPOS tags:

NOUN

Common nouns (default, with PROPN)

PROPN

Proper nouns

VERB

Main verbs – use for action-focused topics

ADJ

Adjectives

ADV

Adverbs

Value

An object of class pos_representation.

See Also

pos_dtm, fit_bertopic, cvalue_representation

Examples

m <- pos_representation(pos = c("NOUN", "PROPN", "ADJ"))

Predict topics for new documents using a fitted BERTopic model

Description

Embeds new documents (or accepts pre-computed embeddings) and assigns each to the nearest topic centroid by cosine similarity. Noise (-1) is never assigned as a target – all new documents receive a real topic.

Usage

## S3 method for class 'bertopic_fit'
predict(object, new_docs, encoder = NULL, embeddings = NULL, ...)

Arguments

Value

A list with:

topics

Integer vector of assigned topic IDs.

probabilities

Cosine similarity to the assigned centroid (proxy for confidence, in [0, 1] for normalised embeddings).

all_similarities

Numeric matrix (nrow = length(new_docs), ncol = n_topics) of cosine similarities to every topic centroid.

Examples

## Not run: 
  enc  <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit  <- fit_bertopic(docs = abstracts, encoder = enc)
  pred <- predict(fit, new_docs = c("new paper about deep learning"),
                  encoder = enc)
  pred$topics

## End(Not run)

Print method for bert_encoder objects

Description

Print method for bert_encoder objects

Usage

## S3 method for class 'bert_encoder'
print(x, ...)

Arguments

Value

Invisibly returns x.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  print(enc)

## End(Not run)

Print method for bertopic_fit objects

Description

Print method for bertopic_fit objects

Usage

## S3 method for class 'bertopic_fit'
print(x, ...)

Arguments

Value

Invisibly returns x.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  print(fit)

## End(Not run)

Print method for bertopic_flow objects

Description

Print method for bertopic_flow objects

Usage

## S3 method for class 'bertopic_flow'
print(x, ...)

Arguments

Value

Invisibly returns x.

Examples

## Not run: 
  enc  <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  flow <- fit_topics_over_time(enc, docs = abstracts,
                               period_var = rep(c("A", "B"), each = 50L))
  print(flow)

## End(Not run)

Print method for hf_classifier objects

Description

Print method for hf_classifier objects

Usage

## S3 method for class 'hf_classifier'
print(x, ...)

Arguments

Value

Invisibly returns x.

Examples

## Not run: 
  cls <- load_hf_classifier("cardiffnlp/twitter-roberta-base-sentiment-latest")
  print(cls)

## End(Not run)

Pretty-print discovered topics

Description

Pretty-print discovered topics

Usage

print_topics(fit, max_topics = 20)

Arguments

Value

Called for its side effect (printing); returns NULL invisibly.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  print_topics(fit)

## End(Not run)

Reassign noise documents to the nearest real topic

Description

Documents assigned to the noise cluster (-1) by HDBSCAN are reassigned to the most similar non-noise topic. Only documents whose best similarity exceeds threshold are reassigned; the rest remain as noise.

Usage

reduce_outliers(fit, strategy = "embeddings", threshold = 0, verbose = TRUE)

Arguments

Value

An updated bertopic_fit.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  fit <- reduce_outliers(fit, threshold = 0.1)

## End(Not run)

Reduce the number of topics by iteratively merging the most similar pair

Description

At each step the two non-noise topics whose centroids have the highest cosine similarity are merged. The loop continues until the desired number of topics remains. After the loop, topics are renumbered 0, 1, 2, ... in order of their (post-merge) IDs and all derived state is recomputed.

Usage

reduce_topics(fit, nr_topics, verbose = TRUE)

Arguments

Value

An updated bertopic_fit with renumbered topics.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  fit <- reduce_topics(fit, nr_topics = 5L)

## End(Not run)

Run a quick Rhobots demo using classic novels from Project Gutenberg

Description

Downloads a selection of classic novels, splits them into paragraphs, and runs the full BERTopic pipeline: embed -> UMAP -> HDBSCAN -> c-TF-IDF. Prints a topic summary and shows two interactive plots: a 2-D topic map and a bar chart of top terms per topic.

Usage

rhobots_demo(n_per_book = 150L, device = "cpu", seed = 42L, verbose = TRUE)

Arguments

Details

The encoder used is sentence-transformers/all-MiniLM-L6-v2 (~22 MB), which is downloaded once and cached by hfhub. Subsequent runs skip the download.

Value

Invisibly, a list with elements fit, embeddings, encoder, and texts (a data frame with columns text and title), so you can continue exploring after the demo.

Examples

## Not run: 
  result <- rhobots_demo(n_per_book = 50L)
  print_topics(result$fit)

## End(Not run)

Check Rhobots system dependencies and print setup instructions

Description

Checks whether the 'torch' C++ backend is installed and prints the appropriate setup instructions. Rhobots requires the 'torch' backend (libtorch + lantern, ~560 MB) to run transformer models. Call this function after installing the package to find out what still needs to be done.

Usage

rhobots_install()

Value

Invisible NULL, called for its side-effect of printing instructions.

Examples

rhobots_install()

Save a fitted BERTopic model to disk

Description

Writes a bertopic_fit object to a directory. The fit is split into:

• metadata.json – human-readable summary (topic labels, counts, parameters).

• fit.rds – the full fit object minus the large matrices.

• embeddings.rds – the document embedding matrix (optional).

• dtm.rds – the sparse document-term matrix (optional).

Splitting the heavy matrices means the directory can be browsed and the metadata inspected without loading gigabytes into R.

Usage

save_bertopic(
  fit,
  path,
  include_embeddings = TRUE,
  include_dtm = TRUE,
  compress = TRUE
)

Arguments

Value

Invisibly, the normalised path.

See Also

load_bertopic

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  save_bertopic(fit, tempdir())

## End(Not run)

Save an embedding matrix to disk

Description

Save an embedding matrix to disk

Usage

save_embeddings(embeddings, path)

Arguments

Value

The resolved file path, invisibly.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  emb <- embed_texts(enc, c("doc one", "doc two"))
  save_embeddings(emb, tempfile(fileext = ".rds"))

## End(Not run)

Measure topic stability across multiple random seeds

Description

Runs fit_bertopic n_runs times with different random seeds on the same corpus, then measures how consistent the document assignments are across runs using the Adjusted Rand Index (ARI).

Usage

stability_analysis(
  docs,
  embeddings,
  n_runs = 5L,
  seeds = NULL,
  ...,
  verbose = TRUE
)

Arguments

Details

ARI = 1 means two clusterings are identical; ARI \approx 0 means agreement no better than chance; negative values indicate systematic disagreement. A mean ARI above 0.8 across all run pairs indicates a stable, reproducible topic structure.

Value

A list of class stability_result with elements:

ari_matrix

Symmetric ARI matrix (n_runs x n_runs).

mean_ari

Mean ARI across all off-diagonal pairs.

per_doc_stability

For each document, the fraction of runs that agreed on the modal topic assignment (1.0 = always the same).

n_topics_per_run

Integer vector of non-noise topic counts.

fits

List of bertopic_fit objects (one per run).

See Also

visualize_stability, sweep_topics

Examples

## Not run: 
  enc  <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  emb  <- embed_texts(enc, abstracts)
  stab <- stability_analysis(abstracts, emb, n_runs = 3L)
  stab$mean_ari

## End(Not run)

Sweep BERTopic hyperparameters and compare topic quality

Description

Runs fit_bertopic and topic_quality over a full factorial grid of UMAP and HDBSCAN parameters (and optionally multiple embedding models), then returns a tidy data frame of quality metrics for every combination. Use visualize_sweep to compare runs visually.

Usage

sweep_topics(
  docs,
  encoders = NULL,
  embeddings = NULL,
  n_neighbors = c(5L, 15L, 30L),
  n_components = c(5L, 10L),
  min_pts = c(5L, 10L, 20L),
  min_topics = NULL,
  ngram_range = c(1L, 1L),
  top_n_terms = 10L,
  extra_stopwords = NULL,
  sample_size = NULL,
  quality_top_n = 10L,
  quality_sample = 500L,
  quality_space = "reduced",
  seed = 42L,
  verbose = TRUE
)

Arguments

Details

Embeddings are the expensive step. When sweeping only UMAP/HDBSCAN parameters with a single encoder, pass pre-computed embeddings so the encoder runs only once:

  emb  <- embed_texts(enc, docs, normalize = TRUE)
  sw   <- sweep_topics(docs, embeddings = emb, min_pts = c(5, 10, 20))

To compare multiple encoders, pass a named list of either encoders or pre-computed matrices:

  sw <- sweep_topics(docs,
    embeddings = list(minilm = emb1, scibert = emb2),
    n_neighbors = c(5, 15), min_pts = c(5, 10))

Value

A list of class topic_sweep with elements:

results

Data frame with one row per parameter combination and columns for all swept parameters plus quality metrics (including n_topics).

best

The selected row of results: highest silhouette among runs satisfying min_topics, or the run with the most topics if no run satisfies the constraint.

min_topics

The min_topics argument (or NULL).

best_met_constraint

Logical: TRUE when best satisfies the min_topics constraint (always TRUE when min_topics = NULL).

n_docs

Number of documents used (after optional sampling).

sampled

Logical: whether a random sample was drawn.

param_names

Character vector of swept parameter names.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  emb <- embed_texts(enc, abstracts, normalize = TRUE)
  sw  <- sweep_topics(abstracts, embeddings = emb,
                       n_neighbors = c(5L, 15L), min_pts = c(5L, 10L))
  visualize_sweep(sw)

## End(Not run)

Compute lexical coherence for discovered topics

Description

Measures how often the top terms of each topic co-occur in the same documents. High coherence means the words form a semantically tight cluster – they appear together rather than in isolation.

Usage

topic_coherence(fit, top_n = 10L, measure = c("npmi", "cv"))

Arguments

Details

Two measures are supported:

"npmi"

Normalised Pointwise Mutual Information, the standard measure in recent topic-model benchmarks. Ranges from -1 (terms never co-occur) to 1 (terms always appear together). Values above 0.1 are generally considered good; above 0.3 is excellent.

"cv"

The C_V coherence measure, a log-conditional variant that is slightly more discriminative on small corpora. Higher is better; typical range -4 to 0.

Value

A list of class topic_coherence with elements:

per_topic

Named numeric vector of per-topic scores.

mean

Mean coherence across all non-noise topics.

measure

Which measure was used.

top_n

Number of terms used.

See Also

topic_quality

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  topic_coherence(fit)

## End(Not run)

Evaluate topic quality for a fitted BERTopic model

Description

Computes four families of metrics that characterise different aspects of topic quality:

Cohesion

How tightly documents cluster around their topic centroid (mean cosine similarity of each document to its centroid; higher is better).

Separation

How distinct topics are from each other (pairwise cosine similarity between L2-normalised topic centroids; lower mean is better, i.e. topics point in different directions in embedding space).

Overlap

How much vocabulary topics share (pairwise Jaccard similarity of the top-top_n c-TF-IDF terms; lower mean is better).

Distribution

How balanced and noise-free the topic assignments are (normalised size entropy, coefficient of variation, noise ratio).

A silhouette score in the full embedding space is also computed. It is the standard cluster-quality measure: values near 1 mean documents sit close to their own centroid and far from the nearest other cluster; values near 0 or below indicate overlapping or mis-assigned topics.

Usage

topic_quality(
  fit,
  top_n = 10L,
  sample_size = 2000L,
  space = c("original", "reduced")
)

Arguments

Value

A list of class topic_quality with elements:

cohesion

List with global (scalar) and per_topic (named vector) mean doc-to-centroid cosine similarity.

separation

List with mean_inter_topic_similarity (scalar) and centroid_similarity (symmetric matrix).

overlap

List with mean_jaccard (scalar) and jaccard_matrix (symmetric matrix of pairwise Jaccard scores).

distribution

List with counts (named integer vector), noise_ratio, entropy (normalised, in [0,1]), and cv (coefficient of variation of topic sizes).

silhouette

List with global, per_topic, sampled (logical), and sample_n.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  q   <- topic_quality(fit)
  print(q)

## End(Not run)

Compute how topic representations change over time

Description

For each unique timestamp (or time bin), the documents assigned to each topic are collected and a local c-TF-IDF representation is computed. Two optional smoothing passes mirror Python BERTopic's behaviour:

Usage

topics_over_time(
  fit,
  timestamps,
  nr_bins = NULL,
  evolution_tuning = TRUE,
  global_tuning = TRUE,
  top_n = NULL
)

Arguments

Details

evolution_tuning

Averages each timestamp's representation with the previous timestamp (L1-normalised) to smooth abrupt changes.

global_tuning

Averages each local representation with the global c-TF-IDF (computed over all documents) so words that are absent from a narrow time window are not completely lost.

Value

A data frame (class topics_over_time) with columns Topic, Words, Frequency, Timestamp, sorted by Timestamp then Topic.

Examples

## Not run: 
  enc  <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit  <- fit_bertopic(docs = abstracts, encoder = enc)
  years <- as.Date(paste0(sample(2015:2023, length(abstracts), replace = TRUE), "-01-01"))
  tot  <- topics_over_time(fit, timestamps = years)
  visualize_topics_over_time(tot, fit = fit)

## End(Not run)

Predict topics for new documents (standalone alias)

Description

Wraps predict.bertopic_fit; see that function for full parameter documentation.

Usage

transform_bertopic(fit, new_docs, encoder = NULL, embeddings = NULL)

Arguments

Value

Same as predict.bertopic_fit.

Examples

## Not run: 
  enc  <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit  <- fit_bertopic(docs = abstracts, encoder = enc)
  pred <- transform_bertopic(fit, new_docs = c("new document"), encoder = enc)
  pred$topics

## End(Not run)

UMAP dimensionality reduction

Description

Wraps uwot::umap. This is the default dimensionality-reduction model used by fit_bertopic.

Usage

umap_reduction(
  n_neighbors = 15L,
  n_components = 5L,
  min_dist = 0,
  metric = "cosine"
)

Arguments

Value

A umap_reduction model object.

Examples

m <- umap_reduction(n_neighbors = 10L, n_components = 3L)

Bar charts of top terms per topic

Description

Produces a grid of horizontal bar charts (via plotly), one panel per topic, showing c-TF-IDF scores for the top top_n terms. The best term is always at the top of each panel.

Usage

visualize_barchart(
  fit,
  topics = NULL,
  top_n = 8L,
  n_cols = 4L,
  width = NULL,
  height = NULL
)

Arguments

Value

A plotly figure object.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  visualize_barchart(fit, top_n = 5L)

## End(Not run)

Interactive heatmap of topic - group associations

Description

Displays the signed chi-square contributions or log_2 ratios from compare_topics as a diverging colour heatmap: blue = over- represented in that group, red = under-represented.

Usage

visualize_comparison(
  comp,
  top_n_topics = NULL,
  max_label_chars = 25L,
  width = 1000L,
  height = 420L
)

Arguments

Details

Topics are placed on the x-axis (with angled labels) and groups on the y-axis, which keeps the chart readable even when topic labels are long.

Value

A plotly figure.

Examples

## Not run: 
  enc    <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit    <- fit_bertopic(docs = abstracts, encoder = enc)
  groups <- sample(c("A", "B"), length(abstracts), replace = TRUE)
  comp   <- compare_topics(fit, groups)
  visualize_comparison(comp)

## End(Not run)

Visualise the hierarchical topic tree as a dendrogram

Description

Produces an interactive plotly dendrogram with:

• Leaf labels: topic ID + top c-TF-IDF words (from fit)

• Internal node hover: merged topics and their pooled terms

• Height axis: cosine distance at which topics were merged

Usage

visualize_hierarchy(
  h,
  fit = NULL,
  n_label_words = 3L,
  width = 900L,
  height = 600L
)

Arguments

Value

A plotly figure object.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  h   <- hierarchical_topics(fit)
  visualize_hierarchy(h, fit = fit)

## End(Not run)

Visualise topic quality metrics

Description

Produces a four-panel interactive dashboard (via plotly) from a topic_quality object returned by topic_quality. The panels are:

1. Silhouette per topic – horizontal bars coloured from red (negative) through yellow (zero) to green (positive).

2. Topic size distribution – document counts per topic, with the noise class shown separately.

3. Centroid similarity – pairwise cosine similarity matrix between topic centroids (lower = more distinct).

4. Vocabulary overlap – pairwise Jaccard similarity of the top-N c-TF-IDF term sets (lower = less overlap).

Usage

visualize_quality(q, fit = NULL, width = 900L, height = 750L)

Arguments

Value

A plotly figure object.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  q   <- topic_quality(fit)
  visualize_quality(q, fit = fit)

## End(Not run)

Interactive heatmap of pairwise ARI scores

Description

Interactive heatmap of pairwise ARI scores

Usage

visualize_stability(stab, width = 550L, height = 500L)

Arguments

Value

A plotly figure.

Examples

## Not run: 
  enc  <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  emb  <- embed_texts(enc, abstracts)
  stab <- stability_analysis(abstracts, emb, n_runs = 3L)
  visualize_stability(stab)

## End(Not run)

Visualise the results of a parameter sweep

Description

Produces an interactive heatmap (via plotly) where each row is one parameter combination and each column is a quality metric. Within every column the values are min-max normalised to [0, 1] so that colours are comparable across metrics (green = best in column, white = worst). Metrics where a lower raw value is better (separation, Jaccard overlap, noise percentage) are inverted before normalisation. Hover text shows the actual raw values.

Usage

visualize_sweep(
  sweep,
  metrics = c("silhouette", "cohesion", "separation", "jaccard", "entropy", "noise_pct"),
  width = 900L,
  height = NULL
)

Arguments

Details

n_topics is always shown as the first column. When a min_topics constraint was passed to sweep_topics, rows that did not meet it are prefixed with "[x] " in the row labels.

Value

A plotly figure object.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  emb <- embed_texts(enc, abstracts, normalize = TRUE)
  sw  <- sweep_topics(abstracts, embeddings = emb,
                       n_neighbors = c(5L, 15L), min_pts = c(5L, 10L))
  visualize_sweep(sw)

## End(Not run)

Sankey diagram of topic flow across periods

Description

Produces an interactive Plotly Sankey diagram where each column represents one time period, each node is a topic (sized by document count), and each link is a cross-period topic similarity above the fitted threshold. Link thickness scales with similarity x min(count_from, count_to).

Usage

visualize_topic_flow(
  flow,
  periods = NULL,
  color_by = c("period", "status"),
  width = 1100L,
  height = 650L
)

Arguments

Value

A plotly figure.

Examples

## Not run: 
  enc  <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  flow <- fit_topics_over_time(enc, docs = abstracts,
                               period_var = rep(c("A", "B"), each = 50L))
  visualize_topic_flow(flow)

## End(Not run)

Visualise documents in topic space

Description

Produces an interactive 2-D scatter plot (via plotly) with one point per document, coloured by topic.

Usage

visualize_topics(
  fit,
  dims = NULL,
  label_topics = TRUE,
  max_label_chars = 30L,
  point_size = 5L,
  noise_color = "#cccccc",
  width = 900L,
  height = 700L
)

Arguments

Value

A plotly figure object, or NULL if no topics were found.

Examples

## Not run: 
  enc <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit <- fit_bertopic(docs = abstracts, encoder = enc)
  visualize_topics(fit)

## End(Not run)

Visualise topic frequency over time

Description

Produces an interactive Plotly line chart showing how the relative frequency of each topic changes across timestamps. Hovering over a point shows the topic's top words at that time.

Usage

visualize_topics_over_time(
  tot,
  topics = NULL,
  normalize = TRUE,
  width = 900L,
  height = 550L
)

Arguments

Value

A plotly figure.

Examples

## Not run: 
  enc  <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  fit  <- fit_bertopic(docs = abstracts, encoder = enc)
  years <- as.Date(paste0(sample(2015:2023, length(abstracts), replace = TRUE), "-01-01"))
  tot  <- topics_over_time(fit, timestamps = years)
  visualize_topics_over_time(tot, fit = fit)

## End(Not run)

Zero-shot topic modeling with user-defined topic labels

Description

Instead of discovering topics through unsupervised clustering, the user supplies a named character vector of topic descriptions. Each description is embedded and every document is assigned to the nearest topic by cosine similarity. Documents whose best similarity falls below threshold are labelled as noise (-1).

Usage

zero_shot_topics(
  docs,
  labels,
  embeddings = NULL,
  encoder = NULL,
  label_embeddings = NULL,
  threshold = 0,
  ngram_range = c(1L, 2L),
  top_n_terms = 10L,
  extra_stopwords = character(0L),
  verbose = TRUE
)

Arguments

Details

The returned object is a full bertopic_fit compatible with all Rhobots accessor and visualisation functions (get_topic_info(), visualize_barchart(), topic_quality(), etc.).

Value

A bertopic_fit object. The extra field $label_names records the original user-supplied label names.

See Also

fit_bertopic, guided_fit_bertopic

Examples

## Not run: 
  enc    <- load_hf_bert("sentence-transformers/all-MiniLM-L6-v2")
  labels <- c(
    "Climate change" = "carbon emissions global warming climate",
    "Machine learning" = "neural network deep learning model training"
  )
  fit <- zero_shot_topics(abstracts, labels = labels, encoder = enc)
  get_topic_info(fit)

## End(Not run)