N-Gram Generator – Word and Character Sequence Frequency Analysis
An n-gram is a contiguous run of n items drawn from a stream of text. Set n = 1 and you have unigrams, the individual words or letters. Set n = 2 and you have bigrams — adjacent pairs such as machine learning. Set n = 3 and you have trigrams. This n-gram generator slides a window of your chosen width across any text and reports every gram it finds together with its count, its share of the total, its rank and the position of its first appearance.
How the generator builds n-grams
The text is first cut into segments. By default a segment ends at a line break and at a sentence terminator, so lazy dog. The quick never produces the nonsense bigram dog the. Each segment is then lowercased and stripped of punctuation according to your settings and split into tokens: words separated by whitespace in word mode, or individual code points in character mode — Array.from is used so that emoji and other astral-plane characters count as one character rather than two surrogate halves.
A window of width n then advances one position at a time across each segment and every window it captures is tallied. A segment of T tokens therefore yields T − n + 1 grams, and the frequency percentage of a gram is its count divided by the grand total across all segments. Turning on cross sentence boundaries merges those segments, which raises the total and shifts every percentage — the counts did not change, the denominator did.
Word n-grams versus character n-grams
Word n-grams capture phrasing. They are what SEO analysts use to find the repeated two- and three-word phrases a page actually emphasises, what editors use to catch crutch phrases like at the end of the day, and what autocomplete and predictive keyboards use to guess the next word. Character n-grams capture spelling and morphology instead. Character bigram and trigram frequencies are the classical entry point for breaking a substitution cipher, they drive language identification for short strings, and character trigram profiles are a standard fuzzy-matching technique for deduplicating names and addresses.
Reading the summary statistics
The type-token ratio is unique grams divided by total grams. A value close to 1 means almost nothing repeats; a value close to 0 means heavy repetition. The repetition score is simply (1 − TTR) × 100, rescaled so a higher number means a more repetitive text. Hapax legomenacounts grams that occur exactly once — in natural language this is typically about half the vocabulary, a direct consequence of Zipf's law, under which frequency falls off roughly as the inverse of rank. Because all three figures depend strongly on document length, only compare them across passages of similar size.
n and the number of possible grams explodes while the number you actually observe barely changes. By n = 4 nearly every gram in an ordinary document occurs exactly once, the type-token ratio approaches 1 and the counts stop carrying information. The comparison view plots this collapse directly. For most texts the useful structure lives at n between 1 and 3.Stop words, padding and Markov probabilities
Removing stop words before building word grams strips out the, of, and and their kin. That is the right move for keyword research, where function words otherwise dominate the top of the table, and it produces a keyword density view of the phrases a search engine would associate with the page. It is the wrong move for linguistic work: function words carry most of the grammatical signal, and removing them creates phrases that never actually occurred.
Enabling padding wraps each segment in <s> and </s> boundary tokens so that sentence-initial and sentence-final context is modelled explicitly — the convention used when training a language model. With padding and n ≥ 2 the Markov table becomes available, converting the counts into conditional probabilities P(next | context) = count(context + next) ÷ Σ count(context + *), ready to seed a Markov chain text generator.
Practical uses
Content teams run an article at n = 2 and n = 3 with stop words removed and a minimum frequency of 2 to surface the long-tail phrases the page genuinely repeats. Students generate training data for a bigram language model and inspect the distribution. Puzzle solvers switch to character mode and turn punctuation stripping off for cipher frequency analysis. Editors sort by frequency to find the phrase they have leaned on eight times in two pages. Everything runs entirely in your browser, so a draft manuscript or a confidential document is never uploaded anywhere.