--- title: "Semantic Topics in AI-Powered Mathematics Feedback" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Semantic Topics in AI-Powered Mathematics Feedback} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include=FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", message = FALSE, warning = FALSE, fig.width = 7, fig.height = 4.5, dpi = 110, out.width = "100%" ) library(sbert) ``` # Why this tutorial The `feedback_translations` dataset bundled with **sbert** contains 8,757 AI-generated feedback messages from a mathematics application, translated into English from ten source languages. Nobody can read nearly nine thousand messages, and many repeat verbatim — "Try again." is the most common, appearing 11 times — so simple frequency lists tell you about templates, not content. Topic modeling answers the question the raw data cannot: *what kind of feedback does the system actually give, and in what proportions?* The answer has to survive scrutiny, so every step here is deterministic: rerunning this document reproduces every number, bar, and sentence exactly. The tutorial first lets the data choose the number of topics, builds the resulting six-topic model, and then inspects it from three deliberately different angles: 1. **Model quality** — coherence and diversity, so you know which topics to trust before interpreting any of them. 2. **Two keyword views per topic** — raw within-topic counts (what the topic *says most*) against class-based TF-IDF (what the topic *alone* says). These are different lists by construction, and the disagreement between them is itself informative. 3. **Representative sentences** — the centroid-nearest messages, which are the auditable evidence that a topic label means what it claims. | Component | Verb | Question it answers | |---|---|---| | Topic count | `compare_topics()` | How many topics does the corpus support? | | Topic model | `topics()`, `fitted()` | What groups exist? | | Model quality | `summary()`, `coherence()` | Which topics are trustworthy? | | Topic size | `topic_sizes()` | How big is each topic, by distinct and reused share? | | Frequent keywords | `terms(sort_by = "beta")` | What does each topic talk about most? | | Distinctive keywords | `terms()` | What does each topic talk about that others do not? | | Evidence | `representatives()` | Do real messages support the label? | Fourteen revision-pinned models are available; this tutorial uses the default, `all-MiniLM-L6-v2` (the field's standard quality-per-megabyte English embedder). The menu, with each model's dimension, input limit, language coverage, and download size: ```{r menu} models() ``` # The corpus, deduplicated Embedding the same string twice wastes computation and — more importantly — lets repeated templates skew the clustering geometry: every copy of a message like "Try again." acts as another point pulling on the same centroid. `dedupe()` collapses the corpus to its distinct non-blank messages (each kind votes once) while keeping the row frequencies, which return as weights when sizes are reported. To keep this tutorial fast and free of any download, it works with the first 600 distinct messages — drop the `head()` to model the whole corpus: ```{r dedupe} corpus <- head(dedupe(feedback_translations$translation), 600) ``` `model_download()` is the one step that touches the network. Called without arguments it fetches the package default, `all-MiniLM-L6-v2` — a 6-layer distilled English model — as its ONNX graph (90.4 MB) and tokenizer into the local cache, locked to one immutable Hugging Face revision and refused unless every byte matches the SHA-256 hashes pinned inside the package, so the model you run today is provably the model you run next year. To use a different embedder, pass its name — `model_download("bge-small-en-v1.5")`, for example — and `models()` lists the other thirteen pinned options. From there `compare_topics()` and `topics()` do the rest: under the hood they encode each distinct message — tokenized, run through the network, mean-pooled with padding masked out, and L2-normalized — into a 600 × 384 matrix, numerically identical to Python `SentenceTransformers` output and bit-identical on every rerun. ```{r download, eval=FALSE} model_download() ``` # Choosing the number of topics There is no universally correct number of topics, and **sbert** deliberately refuses to pick one for you — the right granularity depends on what the analysis is *for*. Instead of guessing, `compare_topics()` fits one model per candidate count and reports the numbers that justify a choice: coherence (do a topic's top terms actually co-occur?), diversity (do topics own their vocabulary or share it?), and the share of embedding variance `explained`. ```{r sweep-show, eval=FALSE} sweep <- compare_topics( corpus$text, n_topics = c(4, 5, 6, 7, 8, 10, 12), measure = "npmi", n_representatives = 5 ) sweep ``` ```{r sweep, include=FALSE} # The same 600 messages' embeddings ship precomputed with the package # (data-raw/feedback_embeddings.R), so the sweep needs no download to build this # vignette; feeding them in is identical to encoding live. .sample <- readRDS( system.file("extdata", "feedback_embeddings.rds", package = "sbert") ) stopifnot(identical(corpus$text, .sample$text)) sweep <- compare_topics( corpus$text, n_topics = c(4, 5, 6, 7, 8, 10, 12), embeddings = .sample$embeddings, measure = "npmi", n_representatives = 5 ) ``` ```{r sweep-print, echo=FALSE} sweep ``` ```{r sweep-plot, fig.height=4} plot(sweep) ``` Coherence is negative throughout — these are short, template-like messages, so any two top terms rarely land in the same short string — but it peaks unmistakably at **six** and falls away on either side, while diversity stays high and `explained` keeps climbing with the count. Six is the granularity this corpus actually supports: the point after which more topics stop buying coherence, and (not by coincidence) about the number of feedback *kinds* an editorial review can act on. `fitted()` pulls the already-fitted six-topic model straight out of the sweep — no re-encoding, no second clustering: ```{r model} topic_model <- fitted(sweep, n_topics = 6) ``` Only `model_download()` and the encoding inside `compare_topics()` are skipped here — the 600 messages' embeddings ship precomputed — so the vignette builds with no download. Run these lines yourself and you reproduce every number below exactly. ```{r model-print} topic_model ``` ## Should you trust these topics? Interpretation comes *after* evaluation, because a beautifully labeled topic with poor coherence is a story about noise. NPMI coherence asks whether a topic's top terms actually co-occur in its messages (+1 = always together, −1 = never); diversity asks whether topics share their vocabulary or own it. ```{r evaluate} summary(topic_model) ``` ```{r coherence} coherence(topic_model, measure = "npmi") ``` Carry these numbers into the per-topic panels below: NPMI rewards regular, repetitive phrasing, so the most template-like topics tend to score highest, and any topic with visibly lower coherence should be read through its representative sentences rather than its keywords. ## Size on two scales The model counts *distinct* messages, but the application sent some messages many times over, so editorial priority follows the weighted share, not the distinct share. `topic_sizes()` reports both scales in one call; the gap between `proportion` and `weighted_share` measures how template-driven each topic is — a topic whose weighted share far exceeds its distinct share is a small repertoire of heavily reused messages. ```{r sizes, fig.height=5} plot(topic_model, type = "sizes") ``` ```{r weighted} topic_sizes(topic_model, weights = corpus$n) ``` # Two keyword views, one topic `terms(sort_by = "beta")` returns the empirical probability of each word given the topic — the *generative* view, dominated by whatever the topic says most often. The class-based TF-IDF scores of `terms()` (its default `sort_by`) are the *discriminative* view: they promote words this topic uses and others do not, and demote words that recur across many topics — frequent, but saying little about any single one. Neither list is "the" keywords; a topic is characterized by the pair. When the two lists agree — as they do for topic 1 below, whose most frequent and most distinctive words are the same ("picture", "pictures", "choose") — the topic has a vocabulary of its own; when they disagree, the counts list is telling you about the corpus and only the TF-IDF list about the topic. ```{r beta} head(terms(topic_model, n = 8, sort_by = "beta"), 8) ``` ```{r terms} head(terms(topic_model, n = 8), 8) ``` # The topics at a glance `plot(topic_model, type = "terms")` shows each topic's distinctive class-based TF-IDF terms, each bar annotated with its score: ```{r plot-terms, fig.height=6} plot(topic_model, type = "terms") ``` `type = "fit"` is the whole per-topic report: all three keyword views — raw within-topic count, class-based TF-IDF, and generative probability (beta) — alongside the topic's centroid-nearest messages, one row per topic: ```{r plot-fit, fig.width = 8, fig.height = 11} plot(topic_model, type = "fit", n_terms = 8, n_representatives = 3) ``` With `per_topic = TRUE` each topic becomes its own figure, the documents stacked beneath the terms — easier to read one topic at a time: ```{r plot-fit-single, fig.height=4.5} plot(topic_model, type = "fit", per_topic = TRUE, topics = 1) ``` The plot text is clipped to fit; for the full, untruncated evidence behind each topic, ask for the representatives directly: ```{r representatives} representatives(topic_model, n = 2) ``` # Where to go from here The model built here is reusable, not just describable. `predict()` assigns any new feedback message to these six topics without refitting; `topic_membership()` replaces the hard assignment with graded probabilities when a message sits between topics. Multi-sentence messages can be modeled sentence by sentence instead of whole: `segment = "sentence"` on the fitting verbs splits every message first and fits the topics on the sentences, and `compare_topics()` takes several levels at once so the unit is chosen from a table and a plot, exactly like the count. The fitted sentence model stores each sentence with its parent message, so `topic_gamma()` returns every message's topic mixture with no further encoding and `topic_sizes(by = "document")` counts messages rather than sentences: ```{r segments, eval=FALSE} comparison <- compare_topics( corpus$text, n_topics = c(4, 6, 8), segment = c("document", "sentence") ) plot(comparison) plot(comparison, type = "fit", n_topics = 6, n_terms = 8, n_representatives = 3) sentence_model <- fitted(comparison, n_topics = 6, segment = "sentence") topic_gamma(sentence_model) topic_sizes(sentence_model, by = "document", weights = corpus$n) ``` When you explore many models on the same corpus — the sweep above, or several term settings — `topic_corpus()` segments, embeds, and tokenizes the messages once and every fit reuses that work, so the exploration costs one corpus pass rather than one per model. For multilingual work on the *source* messages (the `feedback` column spans ten languages), swap one argument — `topics(corpus$text, n_topics = 6, model = "paraphrase-multilingual-MiniLM-L12-v2")` — and the rest of this document runs unchanged.