How to use Remove Duplicate Lines
- 1Paste your list, one item per line
- 2Turn on "ignore letter case" if Apple and apple should count as the same
- 3The de-duplicated list appears immediately
- 4Check the line count to see how many were removed
- 5Copy the result
What Remove Duplicate Lines does
De-duplicating a list looks like one operation but contains three decisions: what makes two lines equal, which copy survives, and what happens to the order. Different tools answer differently, and the answers matter more than the feature itself — a sorted output is useless if the order carried meaning.
Here, lines are compared after trimming leading and trailing whitespace, so an item with a stray trailing space still matches its clean twin. The first occurrence of each is kept, in its original position, and later copies are dropped. The list you get back is your list with things removed, never reordered.
When to use it
- Cleaning an email or contact list
- Merging exports from two systems produces overlap. Case-insensitive matching catches the same address written with different capitalisation, which is the usual form the duplication takes.
- Tidying a keyword or tag list
- Lists assembled from several sources accumulate repeats. Because order is preserved, a list arranged by priority stays in priority order after de-duplication.
- Comparing two lists
- Paste one after the other and de-duplicate: what remains is the union. The drop in line count tells you how much the two overlapped.
- Cleaning log or export output
- Repeated identical lines from a log or a database dump collapse to one each, making the distinct set visible without opening a spreadsheet.
Remove Duplicate Lines settings explained
- Ignore letter case
- Off by default, so Apple and apple are treated as different lines. Turn it on for email addresses, names and tags, where a difference in capitalisation is almost never a real difference.
- Whitespace trimming
- Always applied when comparing. Leading and trailing spaces are ignored, so a line with a trailing space matches its clean twin — the commonest invisible cause of a duplicate that will not go away.
- First occurrence wins
- The earliest copy of each line is kept and later ones are dropped. That means position in the list is decided by where an item first appeared.
- Original order preserved
- Nothing is sorted. The output is your list with lines removed, which matters when the sequence carries meaning — a ranking, a chronology, a set of steps.
What counts as a duplicate
| The two lines | Case-sensitive (default) | Ignore case |
|---|---|---|
| apple / apple | Duplicate | Duplicate |
| Apple / apple | Kept separately | Duplicate |
| apple / apple⎵ (trailing space) | Duplicate | Duplicate |
| apple / apple pie | Kept separately | Kept separately |
Limits and what to expect
- Whole lines are compared — there is no way to de-duplicate on one column of a CSV.
- Only leading and trailing whitespace is ignored; internal double spaces make lines differ.
- Similar is not the same: "John Smith" and "Smith, John" are two distinct lines.
- The output is never sorted, so duplicates that are far apart still both disappear but the survivor keeps its original position.
- Text is not saved — reloading the tab clears the box.
Troubleshooting
- Lines that look identical were not removed
- Something invisible differs inside the line — a double space, a tab, a non-breaking space pasted from a web page, or a different kind of quotation mark. Trimming only handles the ends. Run the text through the text formatter first, which collapses internal runs of whitespace.
- Apple and apple were both kept
- Case matching is on by default. Tick "ignore letter case" to treat them as the same line. For email addresses and tags that is almost always what you want.
- I wanted the last copy kept, not the first
- First occurrence always wins. To keep the last instead, reverse the list, de-duplicate, and reverse it back.
- I need to de-duplicate on one column only
- This tool compares whole lines. For a CSV where only one field identifies a duplicate, a spreadsheet's remove-duplicates feature is the right tool — it can be pointed at a single column.
Your files and your privacy
The list is processed in your browser as you type, with no request to a server. Nothing is stored or logged. That is the relevant property here, because the lists people de-duplicate are usually contact details, customer records or credentials — exactly the material that should not be pasted into a hosted service.
What to use next
FAQ
Does it keep the first or the last duplicate?
The first. Each line's earliest occurrence survives in its original position and later copies are dropped. If you need the last one kept instead, reverse the list, de-duplicate, then reverse it back.
Is the order preserved?
Yes, nothing is sorted. The output is your list with lines removed, in the order you supplied. That matters when the sequence carries meaning — a ranking or a chronology stays intact.
Why were two identical-looking lines both kept?
They differ in something invisible. Trailing spaces are handled automatically, but a double space inside the line, a tab, a non-breaking space pasted from a web page or a curly quote instead of a straight one will all make two lines different. Normalising the text first usually fixes it.
How do I ignore capitalisation?
Tick "ignore letter case". Lines are then compared in lowercase, so Apple, apple and APPLE collapse to one — with the first one you wrote surviving in its original form.
Can I remove duplicates from a CSV by one column?
Not here — whole lines are compared, so two rows differing in any field are both kept. A spreadsheet's remove-duplicates feature can target a single column and is the right tool for that.
Is my list uploaded anywhere?
No. The de-duplication runs in your browser with no network request, and nothing persists after the tab closes.