i-DeClone

AI-ntelligent detection and disposal of clones and near duplicate files


Introduction

Everyone is a digital hoarder nowadays. We take hundreds of photos with our phones, download music and movies, accumulate work documents, and we want to keep it all stored forever. Over the years many file collections are infested with duplicate or near-duplicate items that just waste disk space. i-DeClone will help you discover and reclaim this wasted storage space.

There is no shortage of desktop apps that seek and destroy duplicate files. i-DeClone packs a lot of power under a deceptively simple user interface. It uses Artificial Intelligence algorithms (SVMs) to discover not just identical clones but also near-duplicate files in various categories (Audio/Video/Photos/Documents/eBooks). Near-duplicates are files that may not be identical in byte-by-byte basis, but are useless duplications of information nevertheless, for example:

What are SVMs?
Support vector machines are a class of artificial intelligence algorithms to separate 2 classes of objects, e.g. as similar or not. i-DeClone uses SVMs trained on home-grown data to tell similar files of various types.

Cleaning up your hard disks from unnecessary duplication of information is a complex, time consuming and potentially dangerous operation. You don't want to end up deleting original or unique files. i-DeClone helps you manage this complex task in an easy to grasp and concise GUI, which both novice and computer-savvy users will appreciate. Here are the main ideas:

This document has many internal cross references and links that jump to other parts of information, possibly far away from what you were reading. To come back where you were (before clicking on any anchor), use your browser's Back button or press <Alt+Back> keys.

Font too small to read? Press CTRL+NUMPLUS keys to make it bigger




TABLE OF CONTENTS

Program interface

Scan for duplicates
  Search criteria | Exact or approximate clones? | Advanced search | Duplicate folders | Express mode | Scan progress

Inspect search results
  File details | Preview contents | Compare differences | Revise scan

Mark duplicates
  Mark wizard | Sort options | Focus with filtering | Property filtering | Advanced marking

Clean-up duplicates
  Remove duplicates | Replace with links | Undo mistakes

Managing scan projects
  Saved scans | Periodic cleanups

Program Settings
  Manage lists | Advanced tweaking

Licensing and registration

Miscellaneous

Are you in a hurry? Too bored to read manuals?
Make sure you watch the demo videos (look out for demo video signs)

  Contents  

Program Interface overview demo video: overview of program features

i-DeClone has a sparse and focused user interface, for doing a single job: finding and cleaning up clones and similar files. It resembles a search results pane in windows explorer and similar file managers. This is the main program window showing some duplicate search results:

main window
Figure 1. The main window

Here is a brief description of the main window elements:

Some information and error messages are shown discreetly under the statusbar. These messages go away automatically after a few seconds. If you want to close them immediately just click on [X] button. sample caption message
Expert mode: we said that the results pane is like windows explorer, and it actually is! You can rename with <F2> key, delete items, double-click to open them or use standard shell menu commands like cut and copy. If you want to drag results out of the window, select them and use the right mouse button to drag and drop them someplace else, e.g. a xplorer² scrap pane! Use Open containing folder context menu command to locate items in their folder.

Workflow example

The duplicates scan and destroy workflow goes roughly like this:
  1. Connect all the external devices you want checked for clones and wasted space — or just scan your computer.
     
  2. Click Start scan toolbar button to start a new search or repeat one from the recent history
     
  3. Define the scan parameters: where? (which folders to scan); what? (what kind of files you're looking for); how? (which file properties determine two or more files are clones)
     
  4. Sit back and wait (or do something else) till the search is complete; this part doesn't require your input at all
     
  5. Use Mark duplicates button to tick most duplicates according to your requirements
     
  6. If you are not certain about some of the results (especially if searching for approximate clones), use the preview and comparison tools to bring the results to your liking; manually check or uncheck boxes to override the automatic duplicate marks.
     
  7. If you don't like the results, you can revise the comparison criteria. This is much faster than doing a fresh scan, as i-DeClone won't have to read the files again, only change the comparison method.
     
  8. Once you are happy with the results, click Clean up button to operate on the marked (ticked) duplicates, either delete or otherwise get rid of the space they occupy.
     
  9. When clean-up is complete, quit the program or go back to step #2 for a new search.
What about back-ups?
Sometimes we create duplication of information on purpose, as a safe keeping. Clearly this is not waste of space but essential security for your documents and photos. So there's no point examining your backup drive together with your computer for duplicates. But what if you have the same files in 2-3 copies in your computer? — times 2 if you consider the backups? That's the kind of waste i-DeClone is designed to combat.
These operations will be explained in detail in the following sections.
When set in regular (non-expert) mode, i-DeClone will guide you all the way to step #5 (mark wizard) in a step by step fashion, and you only need to start the cleanup (step #8) manually. In expert mode use the toolbar buttons as necessary to proceed with each phase of the detection/mark/cleanup process. You also get to choose whether you want to see various stage summary dialogs or not. E.g. if you don't want to see New project summary information, tick Don't show this message again box. Such checkboxes are only enabled in expert mode.

  Contents  
Clones can be located anywhere (your hard disk, your phone etc), and could be of any type (music, videos, photos...), but we have to start somewhere. It is possible to tell i-DeClone to look everywhere for everything that's duplicate, but that may cause unnecessary strain to your CPU resources. If you only have duplicate pictures to find, there's no need to search for PDFs or MP3s. If you only store photos in My Pictures folder, there's no need to scan the entire hard disk. So you have lots to gain by specifying where and what to search for.

To focus your search effort, i-DeClone recognizes 5 specific file categories and one all-inclusive:

The last category includes all previous ones, but it is recommended you focus your search effort on one of the specific categories first.

To start a new search click the big Start scan toolbar button. Ignore the project dialog for the time being and click on Start new project button.

Search criteria

A standard windows property page dialog window defines the scan parameters, as seen below. Most people will just use Scan options page, which has the major choices of where and what to search; Advanced page is explained later.
Quick help where is needed
Scan setup dialog is rather complicated. i-DeClone shows balloon popups to help you remember what each option does. Just leave the mouse pointer over a checkbox or other dialog control and an explanation will popup as such:
popup help

scan page file types
Figure 2. Scan options property sheet (and file types detail)

First choose the type of the search via Search for [1] drop-down list, e.g. select Photos to look for images of various sorts. Then decide whether you seek exact or similar duplicates, ticking the relevant radio button in Options [2]. For the time being let's stick to exact clones only. These two basic settings define the file properties i-DeClone will use to compare files to determine whether they are considered clones or not.

Then define the scope [3], that is which folders to search for duplicates. An easy way to add folders is to drag-drop them from windows explorer onto the folders to search box (<Ctrl+V> paste works too). Or use Add toolbar button to choose the folders to search using the standard windows folder picker. You can add just one or as many folders as make sense to your search. To remove a folder, first click on it then use [X] toolbar button or the <Del> key. This type of list and toolbar input method appears frequently in the user interface so it will pay to become familiar with it! For example Save button allows you to save and reuse lists of folders.

Search in phones or not?
i-DeClone can search in a connected Android phone or tablet or any MTP device directly, but frankly the progress will be slow. It will be much faster to remove the phone SD card, insert it in your card reader, and search in it as an external drive!
Any local or externally connected device can be scanned for duplicates, including network drives, but keep in mind that you cannot easily undo deletions from many external folders. If you search in cloud folders (OneDrive, Dropbox, Google drive etc), i-DeClone will not force a download of cloud-only (offline) items. If your cloud files are available locally or your scan options are "easy" (not checking for content) then they can be scanned just like normal files.
If you leave Folders to search list empty, i-DeClone will search in a folder appropriate for the file type chosen, e.g. My Pictures for photos, My documents for documents and so on. If you want to search everywhere, add This PC thispc as the only folder to search and connect all external devices you want checked — phones, external HDs etc
By default all folders you specify will be searched whole, including files in their subfolders. If you switch to expert mode, Search in subfolders checkbox [4] will be enabled; untick it if you would prefer only to examine particular folders without their deeper contents.

If you want to exclude some folders from being scanned, add them in box [3] as usual, and then right click on them and pick Exclude this folder from the menu. Excluded folder names appear in red letters. If you exclude a folder by mistake, use the same menu command to make it "normal" again. The example to the right will scan the entire F:\ drive except for F:\MUSIC and its subfolders.
Note that system or hidden folders are automatically excluded, so you don't need to use this method for those.
excluded

Once you are all set, click Start scan button (or press <Enter> key) to start the scan for duplicate files. Use the project summary to double-check that all scan options are in order.

Exact or approximate clones?

Discovering identical clones is relatively straightforward, there are 2 or more files somewhere in your hard disk with the same size and same content, possibly with same name as well. Things get trickier when we seek files that are "almost" the same but not byte-for-byte clones. For example the 3 photos to the right are not clones but one would consider them "the same thing". Do you want to keep all 3 or perhaps just one would do the trick?

Likewise you can have the same MP3 song encoded in 128 and 256kbps. One file is double in size but the song is the same. Perhaps you want to keep the smallest one, or the largest one if quality is your top concern, but surely there's no point keeping both?
similar pics

i-DeClone can find similar content by various degrees. Just type in the similarity level you seek in scan options dialog [2]. The similarity is measured in percent so 100% would find identical clones, whereas 25% would match many unrelated files as "approximately the same". You may have to experiment with different similarity levels here before you strike a good balance between accuracy and few false duplicates. Revising scans will come handy for fine tuning the similarity level.

i-DeClone calculates a similarity score for any 2 files combining all properties under consideration, either those default for the scan type (music, documents...) or those properties you added yourself. If 2 files are within tolerance for all properties, then they are considered similar and grouped together in the results. If any of the properties differ a lot, then they are dissimilar. As you lower the similarity limit in scan options dialog, more and more files will be considered "tolerably similar". This similarity score depends on the kind of property compared, as follows:

  • Text properties. Text details like Name or Title are loosely compared by bigram (how many pairs of letters match). Items that share many letters are clearly close together, but keyword inversions are also scored high, so JOHN COLTRANE and COLTRANE,JOHN are considered almost the same. JOHN MCLAUGHLIN on the other hand would be far away for most similarity levels.
     
  • Numbers. Number properties like file Size or Length (song duration) are matched with a simple numeric tolerance, for example a file that measures 100 KB is similar to a file sized 90 KB if the similarity level is 90% (or less), but if we enforce 95% similarity they would be considered different.
     
  • Dates. Properties like date Modified or date Taken (for photos) are compared by a fixed scale, where 10 days difference amounts to 1% (this level is tweakable). In this fashion, files modified within 100 days of each other would be considered 90% similar.

Most of the Artificial Intelligence embedded in i-DeClone is for figuring out approximate duplicates. Some quite elaborate algorithms are used, so changing the default comparison properties is not advisable (albeit possible). Different heuristics are used for each major scan category, for example for media files Length (duration) is used to tell which movies are probably the same, Date taken is the major property for photos and so on. You can review the default properties as you change the scan type in advanced page. The good thing is that you don't have to worry about all this complexity, we did all the thinking so you don't have to! :)

Similarity comparisons (without recourse to file content) are only as good as the available tag information in your files (ID3 tags for music, author and title for e-Books etc). If essential tags are missing or they are all set to Unknown artist and TRACK 02, it would be next to impossible to discover similar files. Sadly this is almost always the case with video files, where most tag information is in the filename and nowhere else!
To reduce false duplicates leave the program setting Ignore media siblings turned on; this will ignore approximate music and video files located in the same folder. Alternatively, tick Compare file content option (see below), which makes things slower but more reliable.
If you tweak the properties used for comparison, keep in mind that the first one is very important — it decides the preliminary sorting of files as well as their similarity. So either avoid changing the default first property, or pick one that is fast and representative. Avoid putting text properties first in the list as the similar names NEW FILE.TXT and FILE NEW.TXT are far away in terms of alphabetic sorting. The first property is also enforced more rigidly, it won't go below 95% regardless of the similarity level you choose for the remaining file properties — see n1stColBoost registry tweak.

Content similarity

Some of the search categories, (photos, text documents, music and videos), offer similarity by content if you tick Compare file content box (see figure 3). In this mode we ignore file attributes like name and size/date, and focus on the content only. For documents, only their pure text is examined, ignoring any embedded pictures and formatting (fonts etc). Then you can discover similar "versions" of the same information as for example:
  • Documents with few minor edits (additions/deletions)
  • Photos that are very close in content (e.g. shot consecutively)
  • Photos rotated or resized
  • Songs encoded in various qualities and formats
  • Several versions of the same Movies (e.g. HD resolutions)
As you reduce the similarity threshold in scan options dialog, more and more documents (or photos) will be considered similar enough to be grouped together as "duplicates". Use the preview and comparison tools to assess the quality of the discovered groupings.
Content based similarity comparisons use AI and are probabilistic in nature. So if you execute the same search — even with identical scan settings — twice, chances are that you will obtain slightly different results each time. The similarity level you use is less important in these types of comparisons — items are either strongly identical or not regardless of the percent you set.

The similarity algorithms used for Music and Video scans are more deterministic. i-DeClone actually "listens" to the music and "watches" the movies, extracting mathematical fingerprints that identify a song or video. To make things faster, only the first few seconds of the media file are used for calculating fingerprints; expect a 10% rate of false positives for 90% similarity scans. To reduce such mistakes, increase the similarity percent. If you increase the similarity limit on the other hand, you may miss genuinely identical media that happen to differ a lot in sound quality (e.g. in loudness).

Advanced search demo video: advanced options

If you need more control over the clone detection process, pick the Advanced property page. Here you can manually override the file properties i-DeClone will use for duplicate file detection and tweak some more advanced search options. Below you can see the advanced page controls in normal and expert mode (all controls enabled)

scan page file types
Figure 3. Advanced scan options in normal (left) and expert mode (right)

As mentioned, the choice of file type and exact/similar clones (see general scan options property page) determines the file properties used for comparing files. Usually this means Name/Size and Date last modified in case of exact searches. In other words, if 2 files have the same name, byte size and date, they will be considered identical, and if any of these 3 properties doesn't match, then they are considered different. This covers most but not all cases, hence the availability of advanced tweaks.

Whenever you change the major scan mode (file type and match mode) i-DeClone will adjust the file properties, whether you have Automatic box ticked or not. So first choose the scan mode and then tweak the individual properties.

Here are some reasons why you may need to tweak advanced options:

  • If you want to check file contents tick Compare file content checkbox, to make sure duplicates are exactly the same, byte for byte. Comparing contents is accurate but will slow things down — but see the sidebar note about statistical sampling of file contents. Searching for content implies making sure that file sizes are equal too.
    Scanning by content is sometimes impossible, in which case this checkbox will be disabled.
     
  • You want to find renamed duplicate files, in which case you untick Automatic box, select Name file property in the box and use [X] toolbar button to remove it (this is yet another instance of list and toolbar UI). Then i-DeClone will match duplicates on size and date modified only. In a similar fashion you may want to leave Date out as well and rely on size and content only, in case file dates may have been altered without changing the file content.
     
  • If you want to find music that could be in various encodings (e.g. the same song in MP3 and FLAC format), untick Files must have same extension checkbox. This is usually feasible only in approximate matching, as the binary contents of the audio files in such a case are going to be different. If you tick this option then you should avoid using the full Name to compare files; use Plain name property instead which excludes the extension.
     
  • You want to compare some other file property like Owner, Tags, Comments or any of the hundreds of available windows file properties. Untick Automatic box then use Add toolbar button to insert any available property. In this fashion you could discover relationships among files, not necessarily duplicates, e.g. files that share the same tags. The possibilities are bound only by your imagination!
     
What is content sampling?
To speed up bytewise comparisons, i-DeClone examines the first few MBs of file data exactly, but then switches to a statistical sampling mode. Instead of checking each byte in a 5GB movie, it samples a few bytes here and there. This is almost as exact as a full scan but takes a fraction of the time to establish! Duplicates that were statistically matched show a little X overlay on their icon, signifying the slight uncertainty there is.
overlay

The last two options (ignore system files and links) are always turned on and are disabled in regular scan mode. To override them (at own risk!) activate expert mode.

i-DeClone purpose is getting rid of your personal "junk", without destabilizing your PC. Therefore by default it won't search in system folders like %WINDIR%, installed applications and their AppData, or anything that has Hidden/System attribute. If you want to look into these folders, untick Skip protected system files and folders, but deleting stuff from system folders will do more harm than good.

The last "forbidden" option is Ignore links and small files. The idea is to get rid of waste that takes up lots of space, so very small files and links are not going to help achieving this goal: small files are well, small, and hard/symbolic links are not consuming any space at all. You can untick this option to locate small duplicate files, if it is a big problem in your system, but you will burden the scan a lot for little gain.

Admonitions aside, a final remark has to do with the order of custom file properties used in duplicates detection. In expert mode the file property box toolbar has arrows that move properties up and down in the list. Prefer fast and good discriminating properties like name and size first, then leave slower properties last. This will increase both the speed and accuracy of the detection.

Duplicate folder trees demo video: scan for identical folders

There may be cases where entire folders have identical file contents, including subfolders. Then instead of hunting individual duplicate files it is easier to check for identical folders, and get rid of the duplicates in one cleanup step. Such folders can contain a variety of files (similar to searching for All file types). Scanning for duplicate folders is a complex operation, but the extra effort is well worth it.

folder scan

If you switch to expert mode, the Search for selector (item #1 in figure 2) will include a Folders option. Only exact folder duplicates are supported, as similarity is hard to define in this case. Two folders must have the same name and include the same files (in term of names, byte sizes and dates modified). The same holds for any subfolders, they must match exactly all the way down (including all subfolders thereof). Any file difference discovered at any level will render the folders different (i.e. not duplicate). The recursive scanning algorithm is efficient and well designed, so you should avoid changing the default scanning properties, especially those in Advanced property page.

When comparing files inside each folder, only their name, size and date modified properties are used, in the interest of speed. If you want more robust comparisons including file contents, use nFileSample registry tweak to specify say 4096 or so bytes to compare before two files are considered identical. Note that when even one file pair differs, the entire folder is considered different and dropped from further comparisons.

The algorithm used for folder scanning will consolidate duplicate folders to their highest possible level. For example, say there is a folder C:\TEMP which has a subfolder C:\TEMP\TEST (with several files each), and both folders are identical to another folder structure in E:\OTHER\TEMP (and E:\OTHER\TEMP\TEST), then i-DeClone will mark the only the top folders C:\TEMP and E:\OTHER\TEMP as duplicates, ignoring the subfolders TEST, as these would be cleaned along with their parent folders.

TIP: When comparing particular file types (not folders) and you find many duplicate groups that appear to belong to the same pair of folders, select 2 items, right click and pick Open containing folder menu command. If you have xplorer² this will open the 2 folders side by side so you can compare their entire contents. That could be a hint to switch to duplicate folder scan mode!

Media folders (movies and audio)

Many people store movies and music albums each in its own separate folder, e.g. the sample folder to the right holds the movie 101 Dalmatians and some ancillary files (a readme and SRT subtitles). But the most important file is the AVI movie file itself. If you run a similarity scan for Movies and this particular Dalmatian movie happens to be kept as a duplicate in another folder as well, what you really want to do is to delete the entire folder, not just the movie file. However the 2 movie folder instances may be slightly different in terms of the secondary files they contain (i.e. they cannot be identified as identical duplicate folders with the above method). The menu command Swap with folders, under Mark duplicates toolbar button, exists for such cases. So if i-DeClone has identified 10 duplicate movie pairs, Swap with folders command will exchange the movies with their parent folders, so you can get rid of the accompanying movie files as well. movie folder
If you are searching for duplicate music, the situation is similar, but now you have more files in each album folder. The pic to the right shows a music album with 5 tracks (MP3), and some album art (JPG). If you search for duplicate Music, and assuming the same album exists in another place, i-DeClone will discover 5 duplicate music pairs that correspond to this one album folder. Swap with folders command will replace these 5 groups with just one duplicate folder group (their container folder) music album

It is recommended using mark wizard to tick items to be removed before swapping for the parent folders, because you have richer information on the media file level than on the folder level. For example you can examine file sizes and tick on quality. The swapped folders also have some information (their size and number of contained files), so this isn't a compulsory requirement — if Swap with folders command complains just tick one item and it will let you proceed.

Swapping for parent folders is useful, but the exchange is risky. There are way too many complications that could lead to a disaster, and altough i-DeClone will protect you from the biggest blunders, some situations are too hard to be automatically detected. What if for example, the Maiden voyage music album in the above example has one extra track in one of the 2 almost-identical folders? Or what if a movie folder actually holds 10 unrelated movies and only one of them is duplicated someplace else?
Use swap with parent folders command with caution. There are too many eventualities to consider, and only manual examination of the folder contents can ensure that you are not deleting useful content along with the duplicates. Please select each folder pair with the mouse and press <Ctrl+ENTER> command to open them side by side for inspection. Select any folders that appear to be mistaken and press <Ctrl+DEL> key (or use the right click Remove from pane menu command) to remove them from further processing
split2

Express preset scan mode

Express is the opposite of expert mode: a simple UI experience aimed at beginners. You enable it from settings or by choosing Beginner mode when you first install i-DeClone. It packs all the search functionality of i-DeClone in 3 preset and fixed search types that cannot be adjusted (hence the simplicity).

express scan
Figure 4a. Express scan task dialog

When express mode is active, instead of the detailed scan option dialogs, you get the above simplified version. You can choose the file category (photos, documents, music etc) to search for and one of the preset scans:

  • Quick and exact. Finds exact duplicates of the selected type, using the filename, size and date modified. Usually files that have all these properties common are identical — but note that i-DeClone doesn't check file contents in this "quick" scan.
     
  • Thorough and exact. Finds exact clones considering file content only (implies same file size). It can find duplicates that were renamed too. This is the absolute check for identical files, but checking contents and ignoring filenames make it rather slow.
     
  • Approximate duplicates. This option encompasses all the similarity scan modes at a fixed 90% tolerance (i.e. it finds closely similar as well as identical files). Some types are content checked, and for others it uses tag information (author, title etc).
All 3 scans examine your entire PC and any attached devices (equivalent to choosing ThisPC folder), which covers most eventualities — but it can be time consuming as you cannot target a smaller folder that contains your duplicates. Network folders can be searched if they are mapped to drive letters.

Note that express UI simplifies only the search options. You go through the rest of the procedure (mark duplicates and cleanup) as usual. If you need more control over scan options, click switch to standard mode.

Scan progress

Searching for duplicates takes a long time, especially if you search in many folders and use slow options (e.g. compare file contents). The good news is that you can leave i-DeClone alone to finish its work, while you do something else. The scan task is executed in reduced priority, so your computer should be responsive if you want to do something else on it. Scans have 3 phases:
  1. gathering files: source folders are searched for eligible files, according to your scan options. This part may take long time if you are looking for similar documents and photos by content.
     
  2. sorting files: arranging files on the primary property is very important for obtaining fast results — but sorting itself is so quick that you seldom can see it happening
     
  3. discovering duplicates: items are compared and if they match all properties they are grouped as clones — or ignored if they have no similarities with other files.

Obtaining exact remaining time estimates is difficult (and largely immaterial — you can't make it go faster), so i-DeClone just shows a progress bar as below. Scan progress is also visible on i-DeClone's windows taskbar button. Only phase 3 (discovery) shows a meaningful progressbar, that gives you an idea about the percent of file comparisons completed.

regular progress paused
Figure 4. Scan in progress

Click on the little clock button to pause and resume a scan, if you want to give your PC resources a break. At any point you can abort the scan using cancel button.

i-DeClone will not let your PC enter sleep mode while it's searching. It will not show any error messages or ask your input on any subject whatsoever. You can let it work fully unattended. If you happen to fall asleep in the meantime, your PC will follow suit soon after the scan is complete. You can proceed with marking and cleanup in the next morrow :)

No results found?

If i-DeClone cannot find any duplicate files, it could mean a number of things:
  • Either you don't have any duplicate files to be found
  • Or you searched in the wrong location or/and with the wrong parameters
Assuming that you do have some clones somewhere waiting to be discovered, you can start a new scan expanding the search scope, e.g. search the entire computer (use This PC as the folder to be searched) or/and try different comparison options, e.g. leave the Name property out to find renamed duplicates, reduce the approximate file similarity threshold and so on.

Or congratulate yourself on having a spotless, clone-free storage system and carry on with your day :)

Strategy for fast and thorough de-cloning
To get rid of all your duplicates with the minimum effort, we recommend a 3 step strategy, where you first pick up the "easy fish" so to speak, then you progressively eliminate the more difficult and well hidden duplicates. In all cases you scan your entire PC (starting folder is ThisPC) for any file type you are interested in.
  • Step 1. Do an exact scan for Name/Size/Date file properties, without checking content. This is the fastest and quite accurate mode to get rid of completely identical files
     
  • Step 2. To discover renamed duplicates, do an exact scan for Size including checking file contents. This step is slower
     
  • Step 3. (optional) If you want to remove similar files, pick a 95% similarity level leaving all the default scan options. This will remove the really similar documents only.

  Contents  

Inspect search results demo video: preview clones

Sooner or later the scan will be complete and you will be presented with suspected duplicate files, organized in groups with different background colors. Each group can have two or more related files, and you get to choose which one(s) to keep and which you discard. No automated discovery process is perfect, so you are responsible of double-checking the results, lest you should remove good files by mistake.

Scan results are presented in a window similar to windows explorer, which most people are familiar with. Usually the pane is set in detailed view mode, but 2 other modes are supported, plain list and thumbnails. Switch among these modes using the background menu of the results pane (right click where there's empty space to access the menu), as seen in the picture to the right. To see results in plain list mode tick off both details and thumbnails from the menu. When no columns show, you can check an item's details via the infotip — just hover the mouse over the item for a little bit to see it.
view background menu

Use thumbnails mode if you are reviewing photos. You can automatically make thumbnails bigger or smaller using your mouse scrollwheel while holding down <Ctrl> key to zoom in and out. You can even have combined details with a small thumbnail. i-DeClone automatically turns on thumbnails for picture searches, and turns them off for any other kind of search.
Be careful with i-DeClone search results, they may not be real duplicates. If you tweak the scan parameters changing the default comparison properties, if you don't check file contents, or if you seek approximate duplicates, chances are that some of the files grouped together will be accidentally related and shouldn't be deleted. Please do a thorough revision of the scan results before deleting unique files.

File details

i-DeClone automatically chooses the file details to help you understand the search results. First of all, file properties used for the comparison — either chosen by you or automatically selected — by will be present. i-DeClone will add extra file details depending on the scan mode, e.g. for music searches information like bitrate and genre will also be added, to help you make sense of the results and correct possible mistakes in identification of duplicate groups.

A special column that appears for all scans is % Similar, which gives the similarity score for items in each duplicate group. If you did an exact search, the value is always 100% (identical), otherwise it is a number between the minimum similarity level you specified (usually 90%) and 100 — a higher score means that items are very similar. If many file properties or/and contents were examined, the similarity score is their average.

Another special column is Group # showing the group number. Say i-DeClone found 10 groups of 3 duplicate files each, then each group will be numbered from 1 to 10, and this will be shown in the Group column. Click on group column header to arrange items by groups. Organizing by group is very natural so by default i-DeClone always organizes by group, unless you right click on the pane header and tick off Keep results grouped menu option (see pic to the right). When you click on Name or any other header detail to change sort order, it reorganizes items within each group.

Using the header menu (right-click on Name or any other column) you can also add extra file properties that make sense to your review process, using Select columns menu command. Once you select your favorite details use Set as default menu command to ensure they appear in all scan results.
choose details and grouping

To autosize individual columns, double click on the divider line to the right of the column name. To autosize all columns press <Ctrl+Add> keys or use the autosize command in the header menu. You can reorganize columns left and right dragging and dropping the header with your mouse.
column detail
Figure 5. Choose file details dialog

This is another instance of list and toolbar dialog to manage file details for the results. You can add and remove details, move them up/down in the order, or save and reuse sets of columns, all using the correct toolbar button. When you click on Add button, another dialog comes up that helps you choose a new file property.

Choosing file properties

Modern windows OS offers a rich set of file properties, in fact there are over 1000 to choose from. Not all of them make sense for identifying duplicate files, but there are certainly many interesting ones — including a few extra added by i-DeClone in the Legacy category (users familiar with xplorer² file manager will recognize many of these extra columns, including old school column handlers like TSVN).

choose property
Figure 6. Choose file property dialog

Finding the correct property among hundreds is tricky, but i-DeClone helps:

  • Use the property categories drop-down selector to narrow down the number of choices. Properties are organized for media, image and other common categories
  • Use the property search box with part of the property name you are after, and the list will be filtered to show only properties matching your keyword

The property list shows the type of the property (text, date or number), and a brief explanation of what kind of information the property represents. If you need more information, right click on the property name and use What is this property menu, to find information online.

To conclude the new file property selection, either double click on it, or select it and use OK button to close the dialog.

Preview contents

Use the Preview toolbar button to turn the preview pane (item [6] in figure 1) on and off. It will show a quick preview of the active item in search results. As you change the focused item with the mouse or arrow keys, the preview pane adjusts automatically. i-DeClone can preview photos, text documents, play music and videos and many other file formats as long as you have the necessary preview handlers installed on your system.

You can make the preview pane bigger or smaller using the divider bar. When you place your mouse pointer over the divider bar, it changes its shape to resize. Then you can press the left mouse button to resize it.

Perhaps a more convenient preview method is pressing both left and right buttons simultaneously on a search result item, which generates a large preview of the item under the mouse cursor. Mouse peek preview isn't as rich as the regular preview pane, but it is handy and doesn't take up any permanent screen space. When you release the mouse buttons, the preview goes away.

Compare differences

The preview pane shows one file at a time, but when assessing duplicate groups it is more helpful to compare two files at a time and see their differences. Click on the first file, then hold down <CTRL> key as you click on the second file, and both files will be selected. Then right click on any of the selected files and pick Compare items menu command.

i-DeClone shows differences in text documents, using a free text difference tool called WINDIFF, You can also compare complex document types like PDF and DOCX, in terms of their pure text content (ignoring images and formatting). Thus you get a visual idea of how similar the 2 files are (especially if searching for approximate documents).

Using the same menu command you can compare photos and other documents with thumbnail preview images side by side. For movies you see a filmstrip of 4 video thumbnails spaced by 7 seconds. More file types to compare will be added in the near future.

When you search for whole folders, you can compare their contents side by side using this command. If you have xplorer² installed, the 2 selected folders will open in a dual pane window, ready for comparison. Otherwise, two explorer windows will open side by side.

Compare a few items in each duplicate group to make sure they are indeed clones or similar according to your scan options. To remove items that were not correctly categorized, first select them with the mouse or keyboard then press <CTRL+DEL> keys (or right click and use Remove from pane menu command).

Selecting items in the search results pane works as in windows explorer. If you click on an item it gets selected, and the previously selected items are deselected. To select multiple items, hold down <CTRL> key as you click on them, or use SHIFT+ARROW keys. Preview and context menu commands work with the selected items.

Revise scan parameters

If you feel that the results are not right, or not enough duplicate clones were found, you can always start a new scan — or even better, revise the latest scan project. You will find the revise command under Clean up toolbar button, click on the little arrow to show the menu.

The revision idea is to use the same files already discovered and available in memory with different comparison criteria, so as to match them better. This process is much faster as you won't have to wait to read the entire disc again as in a fresh scan. (acts like a short-term cache).

Revision involves the same property sheet seen earlier only now most controls in the first page (Scan options) will be disabled, as you cannot change the type of files searched for or the search locations — but you can tweak the similarity level. You will concentrate on the second page and try out different properties (leave out Name?), ticking or unticking Compare file content option or whatever else makes sense.

Then click on Start scan to begin the revision. The old results will be discarded and replaced by those matching your revised scan options — and hopefully will be more to your liking! You can repeat the revision process as many times as you like. Some AI similarity scans by content (photos and documents) are semi-random by nature, so you may repeat (revise) them and find slightly different results each time — even without changing any parameters.
Once you begin deleting or removing duplicate results, revision is no longer possible.
Please begin a new scan instead.
revise scan command

  Contents  

Mark duplicates demo video: mark clones manually or automatically

Once the scan results are good, the next hurdle is deciding what to keep and what to discard, which isn't an easy task when many duplicate groups have been found. i-DeClone uses check boxes to mark items you deem duplicate and due for removal. When you tick an item's checkbox, the entire item shows as crossed-out as a visual hint of what's going to happen next.
Selection vs Checked state
You select items in the pane using the usual mouse and keyboard gestures, e.g. click or <Ctrl>+click. Select items to act on them in the short term, e.g. preview or compare them. Selection state is feeble and can be easily lost with a keypress or mouse click. Use the checkboxes to permanently mark items for removal (at a later stage). Checked state is long-term, it is remembered if you filter then show back items. Keep this important distinction in mind! Selected items can be turned into checked/unchecked using <Space> key — or right click menu.

Mark wizard

Sometimes the decision of what to keep and what to mark for deletion is easy and can be easily done using Mark duplicates main toolbar button. For example you may want to keep the most recent version of a similar file (or the original?), keep the best quality MP3 song (larger file size) and so on. Then i-DeClone can easily tick all items in each group except for your preference, and that's the purpose of Mark wizard dialog:

mark wizard
Figure 8. Mark wizard dialog

The various options are fairly self-explanatory. You can only choose one option out of those presented, and this will define which item is kept (unticked) in each group. In particular:

  • As they appear sorted means that the active sort order decides which item to keep (note this method subsumes size and date considerations in the dialog, if the sort order happens to be size or date). So if you sort by path, then you can automatically have the first or last item in each group ticked, if that makes sense. To change sort order click on the column header of interest.
     
  • Keeping items According to location is possible when your scan involved more than one top level folders to begin with (or if you searched your entire computer THISPC folder). You can choose a folder, and any items under this master folder will be spared.
    TIP: leave the mouse pointer over a folder to see its full path
Some groups in the dialog will be disabled if they don't fit with your scan parameters. For example, if you did an exact search that included Size file property, all items in all groups will have the exact same size, so Size and quality group will not be useful as a discriminator and will be disabled.
A shortcut to mark wizard's Keep the first in each group radio button is Keep first in groups command in items context menu. If you want to keep the last in each group reverse the sort order so the last item becomes first!

When a group contains items that are identical in terms of the mark criterion, then the first item in the group will be kept. To avoid such arbitrary choices you can specify additional rules to break up ties.

If you do a mistake and pick the "wrong" mark option, you can always run mark wizard again, and it will do all the checkmarks from scratch. You can also use <ALT+A> keyboard shortcut to unmark all items for a fresh start (see Mark button menu). If you want to mark everything, press <CTRL+A> keys to select everything then press SPACE.

Mark wizard is for choosing the items that remain, which is the natural choice for groups that include more than 2 duplicate items. If you want to do the opposite and mark the items instead of keeping them, first run the mark wizard as if you are keeping the items, then use Invert marked menu command from mark menu. This trick is most useful when you mark according to folder location.
mark button menu

Mark wizard can often sort out all or most of the duplicate groups. If that's not possible you can use mark wizard as a quick first step, then override the pre-checked states and choose what stays and what goes, manually clearing or ticking checkboxes. To manage possibly thousands of duplicate items, you need efficient use of sorting and filtering, as explained next.

Sort options

column header
Click on the column header to arrange by the corresponding file detail, e.g. Name, Size, Path or whatever else shows up as a file detail. Click on the same column again to reverse the sort direction from bigger to smaller and vice-versa. A little triangle shows the active sort mode (Name in the example above). Usually i-DeClone enforces a major sort by Group so you can see the duplicate files bunched together, but you can right click on the sort header and untick Keep results grouped option seen earlier. Note that the group dark/light background color bands will only show if you are sorting by group.

You can <Shift>+click on a column header to set it as secondary (or tertiary...) sort direction. So you can have a primary sort by Path, then shift-click on Size column and have items that have the same path sorted by size inbetween them.

Having items sorted means you can quickly select some of them and either check them or remove them from further consideration. E.g. if you decide that you don't want to deal with files smaller than 100KB, sort by size (after you cancel the auto-grouping mentioned), find and select the first file >= 100KB in size, then press <Shift+Home> to select all smaller files, finally hit <Ctrl+Del> keys (or use Remove from pane context menu command) to remove them from further consideration — they will be safe in their folder and out of your mind.

The right-click menu on items contains many useful commands for selecting many items at once as Select folder siblings that selects items from the same folder, and Select same type that selects based on file extension (useful when you scan for similar mixed types e.g. DOC/PDF). There are many uses of such mass-selection, you can mark (check) them all together in one go, remove them from further consideration and so on.
context menu commands
Right click (context) menu commands operate on selected items, and not on those that are marked (checked) as duplicates. If marked items happen to be selected of course, they will be processed as selected.

Focused marking using filters demo video: mark certain folders

A very useful tool for dealing with search results is the filter bar. If you cannot see it right click on the main toolbar and tick Filter bar menu command. Immediately you get 3 very useful filtering tools:

filterbar
Figure 9. Enabling filtering toolbar
  • Filter results box. What you type in this box is a text filter that shows only items that match your keyword. All the information you see is up for filtering, not just filenames. So you can filter by text appearing in Path or any other column too! (this behavior is adjustable from program settings). Click on the little [X] to the right of the filter box to show all files. Marked files retain their checked status when filtered out then come back into view.
     
  • Color tag filters. Except for checkboxes, i-DeClone uses color tags to mark files. The meaning of this is up to you, as there is no special treatment of colored files, but you can filter out particular colors. So for example, say that you want to make a note of files you want to examine carefully later, first select the lot, then right click on them and assign a color from Color tag submenu (see the earlier context menu snapshot). The meaning of colors is entirely up to you. To see only color marked files use color toolbar button. To show all files press <Ctrl+U> keys or use Show everything from tag filter menu.
    If you cannot see the color tag button, please enable color tags from settings
     
  • Hide groups feature. The next toolbar button has an interesting behavior, it hides all groups that have at least one checked item in them, and then the button remains in depressed state as an indicator of the missing items. The idea is that you process a few duplicate groups and "commit" them out of the way. As you process more groups, hold down <Shift> key as you click on the button (or just click on it twice) to add the newly marked groups and move them out of the way. Thus little by little you progress to deal with more files, while keeping those already handled out of your sight and mind. Then in the end click on Hide button to show all the files, where individual groups will be as you first marked them.
Hide groups button also has a drop-down menu with the commands you can see in the pic to the right. Show marked is a temporary filter that will show just checked items — click on the main Hide marked button to cancel this filter.

The remaining commands implement a "mark clipboard" of sorts. You can store the currently visible checked items and reuse them later, e.g. to add them to a future marked set (Mark stored menu command) or to remove the checkboxes from the stored items later with Unmark stored (in an "invert selection" fashion). Other possible uses of Store marked command include saving the checkmarks against marking mishaps (i.e. undo functionality), remember markings across scans (revised or otherwise) and so on.
mark clipboard menu

No two duplicate screening jobs are equal so it isn't easy to provide commands that automagically "do what you want". However using progressive filtering and your powers of constructive thought, you can achieve complex marking tasks. Note that mark wizard will operate only on files currently showing, ignoring those that are filtered out, so you can apply different mark options to different sets of files (divide and conquer principle).

Here's another use for filtering to mark a set of files:

  • use quick filter box to show items that say, match a folder path
  • Press <Ctrl+A> to select all of them
  • Press <Space> to check/uncheck them all together!
  • (repeat procedure if necessary to mark more files)
Initially the filterbar is on a separate row, but if space permits you can put it together with the main toolbar. Use the drag handle on the left of each toolbar to move them around with your mouse. Or try Auto arrange toolbar context menu command.

Mark and filter using file properties demo video: advanced filtering

filter button
The quick filter box allows simple text-based filtering. For more precise filtering and marking use the filter button next to Hide groups button (or Mark on property menu command). This command lets you target particular file attributes like Path or file size, using rules that apply to the type of file property in question.

filter by column data dialog
Figure 9a. Filter by column data dialog

This advanced filtering dialog lets you pick one of the file details curretly showing in the search results window, and then type a matching rule that will select only the files you need. For example, the setup shown in figure 9a above has the Length (song duration) property selected and the rule >=5:00 will match all songs in the results pane with duration greater than 5 minutes 0 seconds — using the standard mathematical >= symbol to express the relationship. To apply the property rule you have 3 executive buttons that can either:

  • Filter. Show only items that match the property rule, hiding everythig else
     
  • Mark. Automatically tick the boxes for all matching files (mark as duplicates)
     
  • Unmark. Clear the tickboxes for any matching items

This command can act both as a visual filter and as mark wizard. If used as a filter, it works with whatever items are shown, which may be the result of a previous filtering. So you can apply progressive filters with different file attributes (e.g. first on Length, then on Artist) to home in particular files with great precision.

An easy way to cancel any type of filter and show all the duplicate result is to click on the (depressed) Hide groups button on the filterbar   hide groups button

File properties come in 4 varieties, text, number, dates and enumerations (lists). Except for the latter mode, the input boxes are plain text fields where you can type in what you want to search for. The enumerated mode is for parameters like System.Kind where you must choose one of the predefined options from a drop-down list. As you flick through the property drop down list, the input field changes to match the type of the selected property. Leave the mouse over the input field for a while to see a help tooltip with a quick summary of the filtering syntax.

Rules that accept numbers like Size, can be defined in a variety of ways using common mathematical symbols, e.g. all the next inputs are valid:
5 kbmatches files that are exactly 5KB in size
>=4096  search for files at least 4096 bytes large
<=6MBsize up to 6 MB
5 .. 10size in range from 5 to 10 (note use of .. range operator)
~1GBsize approximately 1GB (10% tolerance)
<> 5Not equal to 5

Many numeric properties can have units, for example searching for file size you can either specify 1024 or 1KB which are equivalent. Audio length can be specified in the form MM:SS and GPS coordinates as 145°3'0" (degrees/minutes/seconds). Note some properties are shown in units different to those that are used to find files, e.g. to search for audio bitrate you must enter the number in bps not kbps (128000 vs 128).

The same mathematical operators can be used for date fields, only instead of simple numbers you specify dates, either full with day/month/year (or m/d/y according to your locale) and time, or a partial date like "December 2022" or just "2022". You can also use text descriptors like yesterday and last year, with the aid of the drop-down menu. Here are some date rule examples:

>=30/3/2014    date newer than March 30, 2014
yesterday
2020 .. 2022 date range including the years 2020 — 2022
january 2023 the first month of 2023

You can click on the little clock icon to compose a date if you are too bored to type it!
calendar control

For text rules (e.g. Filename, Path, Artist etc), usually you put one keyword, which can also be a wildcard (*.TXT) or regular expression (TEST[1-9].JPG) — these are automatically identified from their special characters like * or {}. If you put in more than one keyword then all must be present (in any order) for a successful match. If you need to match a whole expression put it "in quotes". For negative keyword match add a leading minus symbol, e.g. THIS -THAT will match documents that contain THIS but not if they also contain THAT.

Text file properties are matched if they contain the keyword you type anywhere, e.g. if you search for .MP3 you will match any MP3 filenames. To match what you type exactly (is instead of contains) use a leading equal sign (=TEST)

If you want to filter on Folder path column, and you want to use a backslash in the search keyword, you must use two of them (eg. DIR\\SUBDIR), otherwise it is mistaken for a regular expression. To turn off automatic regexp detection see binSearchFlags registry tweak. If you are not in expert mode, wildcards are considered verbatim (as plain text)

One possible use of this feature is to mark/unmark pure cloud items, those that are offline (files on demand). These have a distinguishing file attribute called O (for Offline). First make sure that file attribute column is showing for the results (use Choose details command if it's not there), then use the filter dialog to target items with attributes that contain O.

Mark using multiple rules demo video: advanced marking for duplicates

Mark wizard decides which item to keep in each duplicate group using a single property, such as size. In complex situations one property may not be enough; what happens if all items in a group have the same size for example? Multiple rules command (under Mark duplicates toolbar button) caters for just this eventuality. You specify many sequential conditions, and if two items are identical in terms of one property, you will compare them with the next property.

mark priority rules dialog
Figure 9b. Mark priority rules dialog

In a way marking with multiple rules is similar to arranging (sorting) results by many columns. We are trying to find the "winner" item in each group of duplicates which is going to be preserved — all the "losers" will be marked for removal. Use [+] Add toolbar button to add file properties that will be used for comparing group items. You can use any file properties that make sense for the type of scan you are working with (but don't use picture dimensions to sort out MP3 songs!). File properties fall into 3 broad categories:

  • Numbers. Properties like Size or Length (duration) are numbers. Conditions on such properties will favor the largest or the smallest item.
     
  • Dates. Conditions on Date modified or Date taken choose for the most recent or the oldest item in a group.
     
  • Text. When using text properties like Name or Path, you can either try to match a keyword that the "winning" item must contain in said property, or you can choose to keep the longer (or shorter) text, e.g. keep the items that are in shallower folders (shorter Path property)
When you add a new property, you must also select the condition it matches, the "sort order" so to speak (see the Condition column in figure 9b above). There are many ways to specify the condition for each property:
  • Click on the condition column and select what you want from the drop-down list (e.g newest or older date).
  • Press <Enter> or <F2> key
  • Click on Edit toolbar button or use the context menu
A convenient method to change a condition is to double-click on a property name, which will flip over the condition (big will become small and vice-versa).

There are toolbar buttons to delete and reorder rules (moving a rule up in the list with Up-arrow button will make it higher priority). The first rule is the most important, examined first, and usually defines the sort order. If the compared items are identical with respect to this rule, the second property is examined, and so on until there is a "winner". Any leftover rules below the deciding row are ignored. If two items are 100% identical after all the rules are considered, then arbitrarily the first item in the group is kept.

A different way to visualize this multi-rule marking procedure, is to picture the search results, organized by group, then sorting the items in each group using the same properties, where the conditions translate to ascending/descending order. The first item in each group is kept and all lower items in the pecking order are ticked for removal. The sorting analogy breaks when you consider the different handling of text properties though.

Once you have all rules set, click OK button to go ahead and apply them for marking all items for removal except for the "winner".

This command (as well as the simple mark wizard) will discard any items you may have ticked earlier. All visible (unfiltered) items' tickboxes are determined from scratch using the rules defined in Mark priority rules dialog.

If an item happens not to have a property (e.g. a missing date), it is instantly considered a "loser" — unless both items don't have said property, in which case the next rule is considered. If you are searching for keywords in text properties there are 3 possibilities:

  • no item has the sought-after text — rule is skipped
  • ditto in case both items have the keyword
  • only one item has the keyword and it is considered "winner"
Here is an example of using this command. You may be scanning for duplicate/similar songs, and wish to keep those of higher quality (higher bitrate). For duplicate items that happen to have the same bitrate (primary property) you want to keep the one that's shorter duration (less silence at the beginning?). For this scenario you must define these ordered rules:
1. Bit rate is biggest
2. Length is smallest
Note that using byte Size (biggest) would be similar to these 2 rules, but it would select for the Longer item (which would be bigger in size), whereas using multiple rules we can get the finest marking nuances implemented.

If you plan to reuse the same set of marking rules in future projects, use the Save toolbar button and assign them a name. Then you can select the same rules quickly using the Save button drop-down menu.


  Contents  

Clean-up duplicates demo video: delete and undelete clones

i-DeClone has finished its scan and you have examined and ticked checkbox the files that you want removed. Now it's the time to get rid of the marked clones and free up valuable disk space. Click Clean up toolbar button to start the procedure. You have the following options:

cleanup dialog
Figure 10. Choice of clean-up operation

If you click Save results button, you choose not to do anything right now. You save the current scan parameters in a file which you can load later and clean it up. The usual choice is Delete to remove files immediately. In expert mode you have the option to create links instead of deleting clones.

Be careful cleaning up files. Many operations are very difficult to undo, and you may not be able to get back your deleted or linked files. Double and triple check the files marked for deletion before you remove valuable items. If in doubt, back-up the folders before you start deleting.

If there are any items filtered out of view, i-DeClone will tell you about it. You can either work with all files, including those hidden, or just clean-up what you can see, whatever you please! However it does insist that at least one item in each group of duplicates is left unticked; this is the original item that remains or the one that all others in the group link against. If any groups have all items ticked for removal, i-DeClone will show them and ask you to untick one or more per group, before you can continue with clean-up. You can use mark wizard or any other method (e.g. manual unticking with the mouse) to leave one item out per group.

TIP: if you ticked all items in a group because you think that they are all part of a bigger group, then to silence i-DeClone and have them deleted, select all group items to be combined (and one of the preceding group), then right click and use Merge groups menu command. Items will be regrouped so that they all belong to the group with the smallest group ID number. This command is meant to be used for items that were originally assigned to neighbouring groups (eg merge groups 20, 21 and 22) but you are free to use it in any way that makes sense.
Alternatively select one item of the group to be merged and press <F9> key (or use Join groups command from Mark button menu). Then the entire group will be merged with the preceding one.

i-DeClone will then show a summary of the clean-up, which you can accept or cancel. Please study it carefully because it is your last chance to abort before unrecoverable changes begin. Once going, clean-up operations show a progress dialog, which you can pause momentarily or cancel altogether. If any problems are encountered, i-DeClone will tell you about then and ask if you want to continue or abort the clean-up. In the end, all clones are removed from view, leaving behind the unmarked "originals", plus any clones that encountered problems and couldn't be removed. At this point all is said and done, and you can either quit the program or start a fresh scan.

i-DeClone automatically removes orphan group members from view (if a group comprises just one item after processing). So after the end of the cleanup, you may end up with a blank results pane. Don't worry, only the marked clones were removed, the unique unticked files are stored safely in their original locations. This behavior is controlled by Hide orphans setting.

Remove duplicates

This is the natural choice to free up space. Files are deleted in the familiar way (as per your windows explorer or any other file manager), and they end up in the system Recycle bin recycle bin if possible. You will see the usual confirmation, elevation etc prompts as in normal file deletions.

Only local filesystem files and those in USB-connected hard disks are moved to the recycle bin. Other locations are deleted directly and are harder to recover (using backups or undelete tools). Files that cannot be recycled include:

  • Network files
  • Files in USB sticks (flash drives)
  • Files in phones and tablets
  • Deep paths (length >256 characters)
  • All non-filesystem items (e.g. zipfolders, FTP)
Delete confirmations that go into the recycle bin look slightly different than permanent deletions. Please study the following confirmations; the second one (permanent) requires special care because it is very hard to undo.

recycle confirmation permanent removal
Figure 10a. Recycling vs permanent deletion
Files that are recycled are only half-deleted. They still occupy disk space. Once you are sure you won't change your mind, right click on any recycle bin icon, e.g. the one on your PC Desktop, and use empty recycle bin command from the menu. Then the clones will be gone for good empty bin
Linking is a special clean-up function available in expert mode only. Instead of deleting clones, you create a "shortcut" file in the location of each marked duplicate. The shortcut points to the main item of each group (the one that isn't ticked) and most programs cannot tell the difference between a link and the original file. So the clones are still accessible but take up next to no space. Use this option if some of your installed programs expect to find files in their original locations.

There are 2 types of links, each with its own limitations and oddities. Linking is supported for local NTFS regular files only (sorry, no FAT32). If your scan was for approximate files linking isn't possible either — only for truly identical files. In general, if you can afford to, your first clean-up choice should be deleting instead of linking.

  • Hard links. If clones and the main file are on the same hard disk partition (e.g. C:\TEMP\CLONE.TXT and C:\ORIGINAL.TXT), a hard link is preferred. They consume very little space and can be renamed without problems. Unfortunately they are fragile: they can be destroyed (i.e. the linkage information lost) if you edit hard linked files with certain programs.
     
  • Symbolic links. These soft links are more flexible as they can span drive letters, even point to remote network folders. But there are disadvantages too. If you move or rename the real (target) file, then the link is broken and the shortcut can no longer find the file. Even worse, creating symbolic links requires an elevated process (unless your windows is in developer mode). i-DeClone uses a helper clone of itself running as administrator to create symbolic links. You must allow the UAC request to start i-DeClone elevated, otherwise symbolic linking will not work.
Please avoid symbolic links from local to external devices. When the link target is on a removable disk, it will only be useful if you connect the external HD or USB stick, and even then the drive letter could be assigned differently, making the link useless. Always prefer link targets to lie on the same device, or failing that, those stored on your main HD.

i-DeClone will try to create a hard link first, if the situation allows it, as they are more reliable. Failing that it will try a symbolic link. The actual action performed is logged.

If you are working with duplicate folders, i-DeClone will replace one of the folders with a folder junction, that also works as a transparent link to the identical folder contents.

Undo mistakes

Undoing clone deletions is often possible if they ended up in the recycle bin. The trick is that you don't use i-DeClone but your windows desktop or windows explorer! — like a regular file deletion. Immediately after the clean-up is finished or interrupted, right click on your windows desktop and use Undo delete menu command. Your files will be placed in their original locations. i-DeClone may still think the files were deleted, but you can check with your file manager that they were indeed restored.
If you cannot see the undo command in windows explorer menu, then your files were deleted directly and cannot be recovered easily. If you act quickly you may be able to get important files back using one of the undelete tools available, or using file backups. This risk is inherent in all file deletions, and is unrelated to i-DeClone.

If you opted for linking clones, unfortunately there is no undo possible. Theoretically it is possible to use the log file created (see below) and painstakingly reverse all linking commands.
undo delete
i-DeClone creates a log file for each clean-up operation, detailing the actions taken, the scan parameters, any error messages etc. The log is placed in system %TEMP% folder. Logs are text files named DC_CLEANUPxxx.LOG which can open in notepad or any text file viewer. In case of linking, it looks like this (for a sample with 2 groups of duplicate files)
FILES HANDLED ----
C:\Temp\NewCopy.txt
	[S] C:\Temp\NewFile.txt
	[S] C:\Temp\sub\NewFile.txt
C:\Temp\search.ico
	[H] C:\Temp\sub\search.ico
The first file in each group is the main target, which remains intact, and immediately below it follow files that were replaced with [H]ard or [S]ymbolic links to the main file of each group (other possibilities are folder [J]unctions and plain shortcut [L]NK files.) Linked files are indented to the right with tabs. Items with errors are marked with [!] exclamation mark and the error details follow the file path.
You can see the latest cleanup log using Clean up toolbar button menu command View log. These logs are not deleted even when you quit i-DeClone.

  Contents  

Managing scan projects demo video: saved and unfinished projects

When you click Start Scan toolbar button (or press <Ctrl+F> keys) i-DeClone shows a project selection dialog. Here you can click start new project button, or repeat an old scan.

project dialog
Figure 11. Pick a project dialog

The project dialog has 3 lists of projects:

  • Saved projects. Here you will find saved search parameters, which you saved earlier using Save scan toolbar button. You must first complete a scan, then you can use this toolbar button to save its parameters. Please use descriptive names to remember what each saved scan does.
     
  • Recent scan history. List of recent scans. If you cannot identify the scan type from its quick description, let the mouse pointer hang over a list item and a tooltip will appear in a couple of seconds showing full details of each old scan. The latest scan you did is first in the list.
     
  • Unfinished projects. Old scan results you chose to save in a file as part of an old clean-up operation. No new scan takes place, you just see the old results, which you can start processing immediately.
     
  • New project button. Starts a fresh scan for duplicate files where you define all parameters, search locations etc from scratch
A saved scan (parameters) is totally different to an unfinished scan (results). The former is just a set of search parameters and locations, that are used to start a frequently repeated search for clones. An unfinished scan is a half-finished search project which you reload to complete the cleanup of duplicate files.

Each project group box has a count of available items (e.g. Unfinished projects (1) in figure 11). Use the scroll bar/arrows to bring hidden items in view, then click on the desired item to activate it. When you repeat a previous scan for clones (either saved or from recent history), you get to change the parameters if you want to run a slightly revised scan, or just click start scan button to repeat the old search verbatim.

If you no longer need an old project item, right click on it and use Delete item from the menu — or press <DEL> key to remove the selected list item.

Saved results files

If you start a new scan project without first cleaning up the previous duplicates found, i-DeClone will ask you if you want to save the results for later. If you respond with Yes, then a save results IDCS file will be created, and will be available as an unfinished project in project selection dialog.

You can keep such unfinished project files wherever you like, but you could create a new folder for them somewhere (using your file manager) to keep them all together. The latest 5 unfinished projects are easily available through the project selection dialog; if you have more, you can double-click on them and they will open in i-DeClone, the registered program for IDCS files.

i-DeClone will not check if duplicate files were modified since you saved them in an IDCS file. If you nevertheless change the files, some of the duplicate group information may be incorrect! So in general it's a good idea to deal with duplicates quickly, instead of piling up work.

Once you clean up some of the reloaded duplicate files, i-DeClone considers the matter dealt with and will remove the IDCS file from unfinished projects list. Note the IDCS file itself is not deleted automatically — you need to remove these with your file manager.

If you want to export the information of the scan results as they appear in i-DeClone, use the drop-down menu next to Clean Up toolbar button and pick Export CSV text menu command. All the column data will be exported in a standard CSV file and opened in Excel or whichever other program you have to handle CSV files. Groups are identified using Group # column

Periodic clean-ups

Fighting against wasted disk space is a war, not a 1-off battle. Even when you have your PC or laptop relatively clean, duplicates may find their way in, necessitating a fresh scan and cleanup operation. So depending on how you amass new documents, photos, movies and songs, you should periodically re-check your system for similar and duplicate files.

Think of the following scenario: your hard disk is clean, but you want to add some files from an external HD or a network drive. Obviously you don't want to duplicate content that is already present in your local HD. You could do a fresh scan on your local E:\ and the external (say D:\ drive) combined, but that's going to take a long time. Your local E:\ drive is already checked for duplicates, you only need to check its files against the external storage — not within E:\ itself.

For this purpose, when you specify the folders to search in the scan options dialog, right click on E:\ folder and pick Master (clean) folder from the menu. A folder that's declared master is only checked against other folders, exactly what we want in this scenario. Thus i-DeClone will only find duplicates residing in D:\ as quickly as possible.

No scan cache?
Some duplicate cleaners use a cache to speed-up subsequent searches. However the cache file is itself a waste of disk space! A disk-cleanup program that adds its own waste is a contradiction in terms, that's why i-DeClone doesn't utilize a cache.
Note that this isn't the same as checking folder (drive) D:\ on its own; it is checked for duplicates both within itself and against the master folder E:\. For cleanup it makes sense to keep all items under the master folder, selecting E:\ in mark wizard (select it with "Only keep items under this folder").

Master folders are shown in bold in Folders to search box. To cancel a master folder, right click on it and use the same menu command, which acts as a toggle. It is possible to have more than one master folder for a scan.
Master folder contents are read for the scan, but files belonging to a master folder are not compared against each other — only with files residing in other top level search folders — hence the saving in processing time.
paused

  Contents  

Program Settings demo video: navigate program settings

Click on Settings toolbar button (or press <F12> key) to modify various program options. Options are organized by category and you can see what each option does at the bottom of the settings dialog as you choose each option. Click on Defaults button to reset all settings to "factory" values.

i-DeClone normally saves its settings in the registry, but you can have it in a portable mode too, keeping a file called DECLONE.INI next to the program (say if you use it off a USB stick). Use Save settings option (under General category) to define where settings are saved — or if they are saved at all.

If you are not familiar with property controls, they are quite straightforward to use as follows:

  • use up/down arrow keys to move around, or click to select a property
  • start typing to modify text or number fields
  • double click on an item or press <SPACE> to change yes/no options
  • collapse/expand a whole group using [-] and [+] boxes

Here is the complete list of program settings, but note that some of them are not available unless you enable expert mode. Most options are easy to understand, with the assistance of their explanations in the dialog (see the description area [8]).

settings window options part 2
Figure 12. The program settings dialog (showing all options)

Here are clarifications about some of the more tricky options:

  • Expert mode [1]. If you are confident and experienced user, enable expert mode to access all program features, but remember that extra power comes with increased risk of breaking things! You can flip expert mode on/off using the keyboard shortcut <Ctrl+Shift+E> — the program titlebar will show a * if expert mode is enabled.
     
  • Follow junctions [2]. Not recommended. If i-DeClone was to follow special folders that jump to remote parts of the filesystem, chances are that you would discover the same files through two different routes, and although i-DeClone is careful not to delete identical files, it is clearly a waste of time to follow junctions.
     
  • Enter virtual folders [3]. Turn this on if you want to scan into zipfolders, FTP and other special folders. i-DeClone always searches inside MTP devices (phones etc) even if this option is set to NO. Scans will be quite slower if enabled.
     
  • Huge file limit [4]. The limit above which statistical content sampling kicks in. Files larger than the limit are approximately compared and marked with an [X] icon overlay.
     
  • Small file limit [5]. Files smaller than this limit are ignored, if you tick Ignore links and small files advanced scan option. Recommended to put this as high as possible to save on search effort.
     
  • Max stream size [6]. When searching inside phones and special folders (see [3]) this is the maximum size of file content that will be extracted. Files bigger than this will be ignored. This doesn't affect normal files in your hard disc etc.
     
  • File types [7]. The major file types i-DeClone uses are really comma separated lists of wildcards for matching the relevant file extensions, e.g. WAV,MP3,WMA for music category. In expert mode you can amend these list definitions, adding or removing extensions to taste (please separate them with , commas). For All file types category use leading - (minus) to declare extensions to exclude e.g. -DLL,-EXE would search for all files except EXE and DLL.

Manage lists and toolbars

In various parts of the program one comes across a repeated UI feature, of a variable list of folders (or file properties) that is managed by a toolbar. You can add, delete, reorder and save items in the list using the toolbar. For example take the list of file properties used to establish cloned files.

list+toolbar combo
Figure 13. A sample list and toolbar combination for file properties

All toolbar buttons have equivalent keyboard shortcuts, as seen in the toolbar popup help text. e.g. <Del> key would delete the selected list item. You can also right click on list items and manipulate them with a context menu (it offers the same commands as the toolbar).

Double click on an empty space or use Add button to add a new list item. The dialog that ensues depends on the list type, e.g. for folders to scan it will be the standard windows Browse for folder dialog.

Click on Save button to access a menu that can save the current list with a descriptive name of your choice (something that will help you remember what the list is about). Once saved, the list will be available to restore again using the Save button menu. The latest item saved appears first in the menu, under Previously saved ghost item.

To overwrite a previously saved list, modify its contents then save it with the same name. The old definition will be replaced with the new one, and the item will be moved to the top of Save menu.

The procedure to remove a saved item from the menu is a bit quirky: right click (inside the menu!) over the saved item you want removed and use Remove command from the second menu (!). Quite an odd method to delete items arguably, but now you know how to do it :)

Advanced settings tweaking

Some advanced program tweaks are possible through registry editing (or editing the INI settings file). You must first fully quit i-DeClone pressing <Alt+X> keys, then fire up the registry editor typing REGEDIT.EXE in Start > Run dialog (<Win+R> key). Then locate the main registry key HKCU\Software\ZabaraKatranemia Plc\DeClone and change values documented below. It goes without saying that if you are not familiar with registry editing you should stop right here!

EDITING THE REGISTRY CAN RENDER YOUR COMPUTER UNUSABLE.
Unless you are 100% certain you know what you are doing, you are advised not to modify it in any way.

Names that start with "bin" imply values that are made up as the sum of individual bits. Please sum up all the numbers representing the options you want and set this total number to the value. Don't modify any bits that are not documented in the table. An x number prefix indicates hexadecimal (x100 = 256)

Key Value Data Explanation
General binFlags x100 enable searching in very deep folders (overall path >260 letters )
x4000 always scan MTP devices like phones
x80000 write DC_CLEANUPxx.LOG file in %TEMP% folder (includes cleanup actions and any errors)
x1000000 don't use explorer thumbnail cache
binMoreOptions64 disable window alpha-shading when a dialog shows up
x100 favor windows 10 automatic resampler for music fingerprints
x800 prefer LNK shortcuts, then try hard and symbolic links
binSearchFlagsx2000 disables automatic wildcard handling for advanced filtering, search exactly what you type
msDateTolerance ms millisecond tolerance for approximate file date comparisons (default 1000ms *2 = 2.0 second considered identical date, to catch NTFS/FAT32 date differences)
nHours1PCdiffhours How many hours correspond to 1% difference in file dates. Default is 240h which means that files 10 days apart in date modified are considered 1% apart; 2400 hours (=100 days) corresponds to 10% distance and so on.
bin1OffNotify Set this to 0 to reset all the "don't show this again" type of messages and see them all over again
bSantasLittleHelper Set this to 0 to disable background threading (not recommended)
rgbViewBgcol xBBGGRR background color for search results, -1 for the default windows color
szGrepExcludeTypes list Comma separated list of file types (extensions) that you don't want to search for text. E.g. set it to pdf,doc if you don't want to search in PDF and Office documents
szTextViewerPath path Full path for editor you want to associate with <F3> command to view text files (if xplorer² is installed this defaults to editor², else notepad is used)
szFileCountRule wildcard Comma separated list of wildcards for Stock.FileCount property. E.g. *.txt,*.log would count items with txt and log extensions. The default value * counts all the direct subitems in a folder.
nPeekTextWidth pixels Pixel width for peek preview window used for text documents (pictures preview according to their dimensions).
nTextSamplebytes Text characters used for fuzzy document comparison, from the beginning of the file. Default is 64KB of text
nFileSamplebytes Optional number of bytes to examine from the beginning of each file, if not comparing explicitly by content. Default value is 0, which means no contents are compared, only size and date modified.
msSongStartmsec Approximate music comparison by content samples the starting 15 seconds of each audio file. This parameter allows you to vary the duration examined.
msVideoSkipmsec Many movies start with the same "20th century fox" introduction, so i-DeClone skips the first 50 seconds to where the "real" movie starts. If you want to skip more, tweak this parameter
n1stColBoostpercent The first property listed in Advanced scan page is very important and is always set to 95% or higher, regardless of the similarity level of the scan. This registry tweak allows you to change the default, but if you lower it too much the scan times will get very long. You may want to consider increasing it though to e.g. 97% if you get many false positives
nFastFilterCntitems Filtering using all file properties may take a long time, when there are many items shown. This tweak sets the limit of items (duplicates) in the pane above which only fast properties are used for filtering, e.g. Name or Path, but not music Genre — or anything similar that takes too long to extract. Set it to 0 to disable the protective feature.
 
view_e900   nMaxListColumnWidth   pixels Maximum pixel width of list view mode. Set it to 0 for unrestricted size
binMiscOptionsx800000 Don't cross out items marked for removal, just do the checkbox
x100 Highlight the quick filter keyword in matching item details (columns)
 
QuickView binMiscOptions 4 Preview in a background thread for responsiveness
8 Use internal text viewer for all text content
256 Don't animate peek preview window
 
Tag colors name xRGB If you want to add more tagging colors, create this registry key and add the colors as a list (e.g. RED 0x0000FF, GREEN 0x00FF00 etc)


  Contents  

Licensing and registration demo video: How to use your unlock key

i-DeClone is a "try before you buy" application. You can download and install it, then use it for free at full capacity for a limited time period. This way you can tell if it is the right tool for you before buying it. The longer you take to evaluate the more nags it will throw at you, including this nag dialog when you start it:

opening registration nag dialog
Figure 14. Registration prompt dialog

Click on Continue in trial mode to proceed. Eventually after a period of evaluation at full capacity, it will switch to restricted mode. After that the search engine will barely work: it will only show the first few search results. At this point really you should make up your mind, whether to purchase or uninstall.

i-DeClone free trial works full power so you can accurately evaluate the program. All scan options work and all duplicate files will be identified and listed. The main limitation is that you cannot clean too many duplicate files while in trial mode. To unlock the full potential please buy a license key.

i-DeClone is licensed per user. If you are the only user and have 2 computers, you only need one license key. But you must add a seat for each extra user. Your purchase will entitle you to free upgrades and support for one year (at minimum, perhaps longer). Click on the Secure online order button (see figure 14 above) to start the ordering process. All the usual payment methods are available.

Often the program will decide not to nag you for its unlock key during the first day. If you have bought a key and want to use it immediately, press <Ctrl+Shift+L> keys simultaneously, and you will be asked for your license key immediately.

How to use your unlock key

Once you receive your key by email, click on I have an unlock key and start the license manager in full administrator mode. If you see any UAC elevation prompt click Yes to run i-DeClone elevated:

UAC allow elevation
Figure 14a. allow i-DeClone to run as administrator

Open the email with the unlock key, and copy/paste the long funny looking key (all of it, usually 3-4 lines of text) and paste it in the license manager window, then click on Activate license button and you should be able to activate your i-DeClone. If all goes well, you can restart i-DeClone and enjoy the program in full power.

license manager
Figure 14b. Paste unlock key in license dialog

Licensing problems?

If there's something wrong with your key, i-DeClone will tell you about it with an error message, which will include a brief explanation of the problem and a numeric error code (see the pic to the right). Here are the most frequent error codes and their remedies:
  • Key typo (failed checksum -2:3). Did you copy out the entire key? It should be 3-4 text lines and look like the one in figure 14b above. Some people leave out the trailing = (equal) signs. Spaces & formatting doesn't matter but you shouldn't modify the key in any way.
     
  • Key for other program (-3:2). If you own other zabkat programs, you will find that all unlock keys look similar, so you may end up using your xplorer² key to unlock i-DeClone! Please make sure the key includes the token |DECL.1| which indicates a key for i-DeClone.
     
If you still have problems please watch the demo video about using the key, and failing even that you can always contact the support team, quoting the exact error code.
licensing error message

  Contents  

Miscellaneous

Software details
i-DeClone runs on windows Vista or later (7/8/10/11 etc). The installer includes both 32 and 64 bit versions and installs the most suitable for your Windows system.

Customer support
If there is something that you cannot understand using this help file and the demo videos (there are a few under the HELP toolbar button), please contact email support. If you suspect any bugs (e.g. the program crashes) please use the Reliability command from the toolbar help drop-down menu, adding as much information as possible to help us reproduce the problem. You can also ask questions in the forum.

Command line arguments
You can edit the shortcut icon properties (what you click to start the program) to add command line switches and arguments.

COMMAND LINE OPTIONS
/I use settings file (default=declone.ini)
/P run a new process
/G:name start a duplicates search with saved scan project name
/J load all system properties without exclusions (1000+)

A folder name can be passed to start the search where you want. Another way to pick the starting folder is to right click on it and pick i-DeClone Duplicates command from the context menu.

Alternatively, a saved scan results file (*.IDCS) can be passed as the command line argument to start dealing with an old search.

Keyboard shortcuts
Many keyboard shortcuts appear next to the (context) menu commands, e.g. right click on some item or at the background of the file list view. Toolbar buttons also have shortcut keys as indicated in the tooltip, e.g. START SCAN (CTRL+F), where the shortcut key CTRL+F is shown in brackets. Here are some other available commands (similar to xplorer²)

Key Function
Ctrl+A Select all
Alt+A Unmark all
Num * Invert selection
SpaceBar Toggle selected files' checkbox on/off
Alt+Down Jump to next group (UP for previous)
Ctrl+Shift+E  Toggle expert mode on/off
Ctrl+C Copy
Ctrl+X Cut
Alt+Ret Properties
Ctrl+Ret Open selected item's container folder
Del Delete selection (real file delete)
Ctrl+Del Remove selection from pane (files not deleted)
Alt+C Copy selected path names
Ctrl+P Copy selected information (all columns and details)
Ctrl+Plus Autosize columns
Ctrl+Q Toggle the preview pane on/off
Alt+Q Quick popup (peek) preview of the focused item
(press any key to dismiss the preview window)
Alt+2 Preview two items side-by-side for comparison
Ctrl+J Start expert local search mode where only items present (drag-dropped) in the window are checked.
F1 Displays this help file
F12 Program settings
F2 Rename
F3 View selected file as text
F5 Refresh view to clear deleted items
Ctrl+F Start new scan
Ctrl+U Show all files (if anything is filtered out)
Ctrl+wheel Change thumbnail size using the mousewheel

To select multiple items use the middle mouse button. This toggles the selection status without affecting the selection of other items. Another solution is to hold down <Ctrl> key while selecting with the left mouse button.

Press both left and right mouse buttons together to see a popup draft preview of the item under the mouse cursor. Let go of the mouse buttons and the preview window will disappear. The size of the preview window is controlled via program options.


© 2021-2024 ZabKat LTD, All Rights Reserved
zabkat.com