How to remove duplicate lines
A duplicate line is a line whose text matches a line that appeared earlier. The tool reads your text line by line, remembers what it has seen, and keeps only the first copy of each line. The order of the survivors is the order in your original list, so a ranked or hand-arranged list stays arranged.
What counts as a duplicate?
| Option | Default | Effect on matching |
|---|---|---|
| Trim spaces | On | Leading and trailing spaces are removed first, so " apple" and "apple " match |
| Ignore case | Off | When on, "Apple" and "apple" match; the first one is kept as typed |
| Remove empty lines | Off | When on, blank lines are deleted instead of kept as one |
Matching is exact otherwise. "apple" and "apples" are different lines, and so are two lines that differ by a hidden character. If duplicates survive that look identical, run the text through the Text Cleaner first to strip hidden characters and odd spaces.
Duplicate lines versus duplicate words
This tool works on whole lines. It does not remove a repeated word inside a sentence, such as "the the". To count repeated words, use the Word Frequency Counter. To remove a repeated word, search for it with Find and Replace.
Remove duplicates and sort in two steps
Often you want a unique, alphabetical list. Remove duplicates first, then paste the result into the Alphabetizer. Doing it in this order keeps the sort fast and the counts easy to check.
Limits
The tool deduplicates lines of pasted text only. It does not open files, and it does not compare columns in a spreadsheet, so for a table, copy the one column you need. It does not merge near-duplicates or fix spelling variations. Very large lists are limited by your device, not by a fixed cap, so split a huge file into parts if your browser slows down.