Data Loader can deduplicate records while importing them.
Salesforce Data Loader does not remove duplicate records during an import. Duplicate entries in the input file must be identified and removed before the load; duplicate record IDs can cause a Duplicate Id Specified error or unintended repeated updates.
Duplicate Id Specified
Community Discussion