Duplicate Line Remover
Delete repeated lines in one paste — order kept, counts shown.
Or drop a text file on the box. Nothing leaves your device.
Options
Most repeated lines
| Line | Times | First seen |
|---|
About the Duplicate Line Remover
Paste a list — email addresses, keywords, URLs, product codes, log lines — and every repeated line is removed as you type. The first copy of each line stays exactly where it was, so the order of your list does not change. You can also keep the last copy instead, list only the lines that repeat, or only the lines that appear once.
Comparison can ignore upper/lower case and spaces at the ends of lines, so Apple, apple and APPLE count as one line when you want them to. Counts show how many lines went in, how many came out and which lines repeat most. Large lists are fine too: removing duplicates from a million lines takes a fraction of a second on a typical laptop, entirely in your browser.
How to use it
- Paste your text into the box, or use Open file (or drag a file onto the box) for a
.txt,.csvor.logfile. The file is read on your device. - Choose how lines are compared: Ignore case, Ignore spaces at line ends, and what to do with Blank lines.
- Pick which copy to keep — First copy or Last copy — and what to Show: each line once, only the lines that repeat, or only the lines that appear once.
- Copy the result or download it. Switch to Removed lines to see exactly what was taken out.
Examples
seo tools SEO Tools keyword planner seo tools rank tracker
seo tools keyword planner rank tracker
5 lines in, 3 lines out, 2 duplicate copies removed.
INV-1001 INV-1002 INV-1001 INV-1003 INV-1002
INV-1001 INV-1002
react vue react svelte vue
react svelte vue
Each line moves to the position of its last appearance.
Common uses
- Cleaning email or phone lists before importing them into a mailing tool or CRM.
- De-duplicating keyword research exports, hashtags and URL lists.
- Finding repeated invoice numbers, order IDs or SKUs with Only lines that repeat.
- Reducing a log file to its distinct messages, with a count of how often each one occurs.
How lines are compared
Two lines are duplicates when they are identical after the options you choose are applied:
- Ignore case treats
AppleandAPPLEas the same. It also matches letters that have more than one upper- or lower-case form, such as GermanSTRASSEandstraßeor GreekΟΔΟΣandοδος. - Ignore spaces at line ends ignores spaces and tabs at the start and end of each line. The line you keep is output exactly as it was written.
- Characters that look identical but are stored differently — for example
étyped as one character or asefollowed by an accent — are always treated as the same.
Everything else must match exactly: colour and color are different lines, and so are résumé and resume.
Blank lines
- Keep in place leaves empty lines where they are, so paragraphs and groups stay separated. They are not treated as duplicates of each other.
- Remove deletes every empty or whitespace-only line.
- Treat as a line handles blank lines like any other line, so only one of them is kept.
When you show only repeating lines or only lines that appear once, uncompared blank lines are left out.
Command-line equivalents
If you know Unix tools: each line once with the first copy kept is the same as awk '!seen[$0]++', only lines that repeat is like sort | uniq -d, and only lines that appear once is like sort | uniq -u — except that this tool keeps your original order instead of sorting first.
Limitations
- Lines are compared as whole lines. To find repeated words inside a line, put one word per line first.
- Near-duplicates (typos,
LtdvsLimited) are not detected — only lines that are identical after the chosen options. - Files up to 50 MB can be opened. Very large results are previewed on screen, while Copy and Download always include the complete text.
- Files are read as UTF-8 or UTF-16 (as saved by Windows PowerShell or Notepad). A file in an older Windows encoding is read as Windows-1252 and a note says so; files in other legacy encodings should be re-saved as UTF-8 first. Results are always saved as UTF-8.
Privacy
Everything happens in your browser. What you enter or open here is not uploaded or stored by MySmartCoPilot.
Frequently asked questions
Does it keep the original order of my list?
Yes. Lines are never sorted. With First copy each line stays where it first appears; with Last copy it moves to where it last appears. If you want the result in alphabetical order, use the Text Sorter.
Why were two lines that look the same not removed?
They differ in a way you cannot see: letter case, a space at the end, a tab instead of spaces, or an invisible character such as a zero-width space or a non-breaking space. Turn on Ignore case and Ignore spaces at line ends, or clean the text first with the Text Cleaner.
Can I see how many times each line appears?
Most repeated lines lists the lines with the most copies and their counts. Download counts (CSV) gives a complete frequency table of every distinct line that you can open in Excel or Google Sheets.
How many lines can it handle?
It is built for large lists: removing duplicates from one million lines takes a fraction of a second on a typical laptop. With text that large, most of the wait is the browser displaying it in the box. You can open files up to 50 MB.
Are Windows line endings kept?
Yes, for files: when you open a file with Windows (CRLF) line endings, the copied and downloaded result uses CRLF too. Text typed or pasted into the box uses the browser's standard line breaks (LF).
Is my list uploaded anywhere?
No. The comparison runs in your browser and nothing you paste or open is sent to a server, so it is safe for customer lists and other private data.