Remove Duplicate Lines

How it works

Paste your list, one item per line. The tool reads it top to bottom, keeps a line the first time it sees it and drops every later line that matches. The counts under the result show the total, unique and removed lines.

Each line is turned into a comparison key before it is looked up, in this order:

1. Surrounding spaces (if Ignore surrounding spaces is on): spaces, tabs, non-breaking spaces and the zero-width characters U+200B, U+200C and U+200D are removed from both ends. The kept line is also output without them.
2. Unicode form (if equivalent forms are treated as equal, on by default): the key is normalised to NFC, so "ş" as one character matches "s" followed by a combining cedilla, and "ü" matches "u" plus a combining diaeresis.
3. Letter case (if Case sensitive is No): the key is lower-cased with the rules of the selected language.

Only the key is changed. The output is the original text of the line that was kept. With Last occurrence the scan runs from the bottom, so the last copy of every line survives, and the list is then put back in order.

Line breaks of any kind (Windows CRLF, Unix LF, old Mac CR, and the Unicode line and paragraph separators) end a line, and the result is joined with plain LF. One break at the very end of the text does not create an extra empty line.

Examples

Why the casing language is a setting. In the Turkish rule "I" is the capital of the dotless "ı" and "İ" is the capital of "i". So "İstanbul" and "istanbul" are the same word, while "ISTANBUL" lower-cases to "ıstanbul", which is a different key. Under the English rule "I" becomes "i", so "ISTANBUL" matches "istanbul", but "İ" turns into an "i" with a combining dot and no longer matches. Pick the rule of the language your list is written in.

Language-specific limits. Lower-casing is not full case folding. In German, "STRASSE" and "Straße" stay different lines, because ß is not turned into "ss", and a Greek final sigma "ς" does not match "σ".

Things to watch for

• Invisible differences are the main reason duplicates survive. Lists copied from web pages, PDFs, spreadsheets or chat often carry non-breaking spaces, zero-width characters or trailing tabs. Two such lines look the same on screen and compare as different. Turn on Ignore surrounding spaces first when a result looks wrong.
• Not every invisible character is ignored. Only whitespace and the three zero-width characters at the start and end of a line are trimmed. A soft hyphen, a word joiner, left-to-right and right-to-left marks, variation selectors, or any invisible character inside a line still make two lines differ.
• Normalisation is canonical only. "ş" written two ways matches, but the ligature "fi" and the letters "fi" do not, and neither do full-width and ordinary letters.
• Trimming rewrites the kept lines. With Ignore surrounding spaces on, the output lines have no leading or trailing spaces, including blank lines that consist only of spaces, which come out empty.
• Empty lines count as duplicates only if you ask. Without trimming, "" and " " are different lines even when Keep empty lines is No.
• Line endings are normalised. A file with CRLF endings is written back with LF. If the target program needs CRLF, convert afterwards.
• Sorting is a separate step. Removing duplicates keeps the original order; it does not group or sort anything.
• Size. Up to 500,000 characters are accepted; the comparison itself is a single pass, so even long lists finish immediately.
• Your text stays on your device. Lines are compared in the browser, nothing is uploaded or saved.