Shulou(Shulou.com)06/03 Report--

According to the MS website, deduplication is the process of finding and removing duplicates from data without compromising data fidelity or integrity. Store more data in less space by segmenting files into small variable-size blocks (32-128KB), identifying duplicate blocks, and maintaining copies of each block. Redundant copies of a block are replaced by references to a single copy. Compress blocks and organize them into special container files in the System Volume Information folder.

After deduplication, files are no longer stored as separate data streams, but instead are replaced with stubs pointing to data blocks stored in the common block store. These file share blocks are stored only once, reducing the disk space required to store all files. During access to the file, the correct blocks are transparently assembled to provide the data without invoking the application or user knowing the disk translation to the file. This enables administrators to apply deduplication to files without worrying about any changes to the application industry or the impact on users accessing those files.

The diagram above is an example of storage without deduplication enabled, and the following is how files are stored with deduplication enabled.

Server and volume requirements for deduplication:

1. Server hardware must meet the minimum requirements for running Windows Server 2012, such as memory not less than 4GB.

2. If you are running deduplication on multiple volumes on the same server, proper hardware planning is required. A volume requires 1 CPU physical core and approximately 350MB of dedicated memory. It can process about 100GB of data per hour or about 2 terabytes per day. Data deduplication uses 25% of system memory in background processing mode. For example, if you have a 16 gigabyte system, the memory available for deduplication is 4 gigabytes, so the memory can support 11 volumes for deduplication.

3. Data deduplication supports up to 90 volumes at a time;

4. Data deduplication cannot be applied to system or boot volumes. Partitions can be MBR or GPT partitions and must be formatted as NTFS.

In this article, we'll take a brief, experimental look at how to achieve deduplication and restore file access in the event of a disaster.

1. Install the deduplication module

Enable deduplication for a partition

Open Server Manager, locate File and Storage Services below, select a volume on the server, and configure deduplication on the volume.

Run deduplication, we will deduplicate files older than 0 days.

In the lab environment, Volume E has five identical folders, occupying 3.06GB of space before deletion.

For non-system volumes, boot volumes, or volumes with deduplication turned on, you can also run an assessment tool to see how much space can be saved. In this lab, we copied these five folders to F:\data, and evaluated the space savings of deduplication on this directory.

Manually trigger a delete.

Start-DedupJob -Volume E: -Type Optimization

After the deletion is completed, the entire space occupied by E disk is only 800 megabytes, while the previous five folders only occupied 3.06 gigabytes.

We can see the effect of redeletion by using the following command get-dedupstatus| format-list

Then, let's take a look at these five folders, the space occupied is 0, that is, these files are not stored in the location we see, but placed in the System Volume Information head.

At this point, we can see that it is indeed saving 2.45GB of storage space, saving nearly 80% of the space, which also confirms that the same data is stored only once. But windows compares files by blocks, not by underlying blocks.

Next, let's look at how, one day, the OS crashed, we reinstalled, and how to make the data accessible.

To verify that the file is accessible, I'm going to copy a file from the folder where deduplication was run on the reinstalled 2012. It turns out it can't be copied.

At this time, we should reinstall the role of data deduplication on the server of the newly installed operating system. After the installation is completed, the relevant rules for deleting the original E disk volume will inherit those before reinstallation. No additional settings are required, so the data can be accessed normally again.

Attached: Microsoft deduplication link: https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/hh831602(v=ws.11)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.