find and delete duplicate files with just Powershell

... analog computer!

Consider this. You have the same files with different file names spread out over a bunch of folders. If you are on a recent Windows machine, Powershell is all you need to get out of that mess and delete the duplicates.
This also means you get to do this from the command line which makes it extra l33t.

Cool. Let’s get started.

ls *.* -recurse | get-filehash | group -property hash | where { $_.count -gt 1 } 
| % { $ | select -skip 1 } | del

You’re done.

Alright. Here’s what going on in detail:

ls *.* -recurse             # get all files in the current folder and all subfolders
get-filehash                # calculate the file hash for all files
group -property hash        # group them by the hash value
where { $_.count -gt 1 }    # select those groups with more than 1 item
% {                         # for all groups
    $ |              # output the group content, which are the files
    select -skip 1          # select all but the first file 
   }                        # (we don't want to delete all of them right?)
del                         # delete the file

If you want to experiment with this I’d recommend you to change the last del command with something safer, like echo which just prints out the file or by adding the -WhatIf parameter so simulate a delete.

Oh yeah, **DISCLAIMER**. Don’t just randomly copy past Powershell code and execute it on your machine if you don’t know what you are doing. Certainly if it’s deleting files like the example above. You might end up deleting more than you bargained for.  :)

Photo by James Vaughan, cc-licensed.

33 thoughts on “find and delete duplicate files with just Powershell

  1. n3wjack Post author

    Thanks for posting the final fix for this! It will help other people looking to do the same thing.

  2. Lexis

    i fixed it with this

    ls *.* -recurse | get-filehash | group -property hash | where { $_.count -gt 1 } | % { $ | select -skip 1 } | % { move-item -path $_.path -destination C:\yourDuplicatePath }

  3. Lexis

    here is my final code
    ls *.* -recurse | get-filehash | group -property hash | where { $_.count -gt 1 } | % { $ | select -skip 1 } | % { move $_ C:\dups }

  4. Lexis

    I would like to move the files instead of copy. but when i replace the move with del i get this error

    move : Cannot find drive. A drive with the name ‘@{Algorithm=SHA256; Hash=8F31F6492F4A11567BDE5A1A0553E16E179E6DBBAA5CD618DB2515585E134418; Path=C’ does not exist.
    At line:1 char:122
    + … unt -gt 1 } | % { $ | select -skip 1 } | % { move $_ c:\dups }
    + ~~~~~~~~~~~~~~~
    + CategoryInfo : ObjectNotFound: (@{Algorithm=SHA…E134418; Path=C:String) [Move-Item], DriveNotFoundException
    + FullyQualifiedErrorId : DriveNotFound,Microsoft.PowerShell.Commands.MoveItemCommand

  5. n3wjack Post author

    If you want to move the files, you need to replace the last delete statement with a loop, and move every file passed into the loop.
    So that ends up looking like this (I’m leaving out the first bit of the statement):

    ... | % { move $_ c:\yourfolder }

  6. Iswar

    I would like to do the same as the above script, instead of deleting move the duplicates to another folder.

  7. Yarin

    Thanks for that! Proved very helpful when I tried to finally do a backup of all my image files from the last 20 years, which were scattered across hundreds of CDs, HDDs and USB sticks, with a TON of duplication.

  8. Pingback: Поиск дубликатов файлов Windows |

  9. TheAngos

    If you want to experiment with this I’d recommend you to change the last del command with something safer, like echo which just prints out the file.

    Another option would be to use the -WhatIf parameter on the del, which is an alias of Remove-Item. You get this instead of a deleted file:

    What if: Performing the operation “Remove File” on target “C:\duplicatefile”.

    If the result looks correct you can then remove the parameter and let it execute and actually remove items.

  10. jacob2017

    Hello Good day, I use a software called Duplicate Files Deleter, it’s very easy to use and after it finds the duplicate files it lets you chose what you want to do with them (copy/delete/move). You can even check network files and you can check multiple paths in the same scan. This helps me a lot. I hope you too.

  11. protogon

    Huge thanks for this script! Now I’m finally able to finish my Spotlight Grabber – a tool that grabs and sorts Windows Spotlight wallpapers, and now even removes duplicates.

  12. n3wjack Post author

    The get-filehash makes you loose most of the file information, except the Path. But you can get that back by doing another ls using the path, and then write whatever file properties you want to your CSV file.
    Adding the following instead of the del statement should do the trick:

    .. | % { ls $_.path } | select name,length | export-csv c:\temp\dupes.txt -NoTypeInformation

  13. Markus


    well done – as mentioned earlier, ist easy to use and nothing needs to be installed on the host.
    I was looking for something to analyze a filetree only, therefore I’ve altered the last pipe from “del” to:
    select Path| Export-Csv C:\temp2\duplicates.txt -NoTypeInformation

    Some of you may see a way to get the filesize in the csv, too?

  14. n3wjack Post author

    You have to paste the whole command in a single line to make it work. If there is a return in the middle you’ll get errors like this.
    If you copy paste it from the browser you’ll probably have 2 lines instead of one. Just paste them into a text editor first and put everything on a single line.
    Oh yeah, and be sure to run it without the delete statement at the end first, so you’re sure it won’t delete anything you don’t want it to. :)

  15. Edward Norton

    I get an error.. what’s up?

    At line:2 char:1
    + | % { $ | select -skip 1 } | del
    + ~
    An empty pipe element is not allowed.
    + CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : EmptyPipeElement

  16. Pete Spence

    Thanks for this, I was looking for a way to get unique values (ordered chronologically descending) from a hash where duplicates existed and the select -skip 1 will work a treat :-)

  17. Devin

    Super slick one liner! I ended up taking out the “select -skip 1” and swapped the “del” for an “ls”. Piped it to a CSV. Clean up the output a little bit and I can hand this to an end users so they can review why they are eating up a ton of space on my SAN.

  18. n3wjack Post author

    Good idea. For a large amount of files that would indeed speed things up I reckon.
    In my case the statement I used was fast enough to do the job for the amount of files I had to process, so I didn’t have any need to optimize the statement.

  19. Carnino

    I would suggest that you first group by file size, then filter for groups with more then one element and only then run the file hashing. That will be much more efficient and WAY faster than having to hash every single file.

  20. Herschel

    Thanks for posting this. I don’t think I’ve used the “group” feature of powershell yet. Pretty nice.

  21. n3wjack Post author

    I think you missed the point here. With that single line of PowerShell you can delete duplicates without even having to download or install any additional software.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.