Shulou(Shulou.com)06/03 Report--

Background

Due to historical reasons, the file data is stored in two sets of FastDFS environments. Due to the project requirements, the two environments need to be combined into one, and a unified interface is needed to access the data. Check and find that the Key generated by the two sets of FastDFS uploads is different, so there is no need to consider the file conflict here.

It is simply verified that data merging can be achieved by copying the storage of environment B directly to environment An and accessing it through the domain name of environment A (FastDFS accesses data through Nginx, so as long as there are files on the path pointed to by Nginx, access is not a problem). However, this brings a problem. When FastDFS expands and adds nodes later, these copied files are not recognized by the tracerServer of environment A, so this part of the data may be abnormal after expansion.

Environment information environment An address 192.168.1.49 file key is wKgBMV. Environment B address 172.18.73.129 file key is rBJJgV... Environment An added server address 192.168.1.50 (for migrating data from environment B) extended server address 192.168.1.51 for environment A (used to verify cluster expansion after merger) implementation principle

A new storage_id function has been added after FastDFS4.0, which can specify an id alias for each storage node. When storage is migrated, if the migrated IP is inconsistent with the original service IP, you only need to change the direction of the ID to complete the migration. This time, since storage_id is not enabled in both environments, you only need to enable it in the two environments and assign different id to them (id cannot conflict). Then add a new node to environment A, and the ID used by the new node is the id already specified by environment B, and then put the data of environment B on the new node, so that the two nodes will automatically synchronize the data with each other and merge.

If storage_id is enabled and id conflicts, it is assumed that you only need to expand the capacity once, assign a new ID, migrate the data of the new ID, and then merge it.

Operation process one, open storage_id1. Modify / etc/fdfs/tracker.conf#use_storage_id = falseuse_storage_id = true to assign 100001 of ID to environment A (192.168.1.49) and 100002 of ID to environment B (172.18.73.129) 2. Modify the environment A / etc/fdfs/storage_ ids.confs [root @ localhost fdfs] # cat / etc/fdfs/storage_ids.conf# 100001 group1 192.168.1.4 group1 100002 group1 192.168.0.1163. Modify the environment B / etc/fdfs/storage_ ids.confs [root @ localhost fdfs] # cat storage_ids.conf# # 100001 group1 192.168.0.196 100002 group1 172.18.73.129 restart the service and check the switching

Second, output the MD5 information of two sets of environment files for verification. # File summary of generation environment A [root@localhost] # find / data/fastdfs/storage/data/-type f-print0 | xargs-0 md5sum > / file summary of tmp/1.49.md5# generation environment B [root@localhost ~] # find / data/fastdfs/storage/data/-type f-print0 | xargs-0 md5sum > / tmp/73.129.md5 3. Package two sets of Storage for the environment respectively. # backup environment A data, which can be used to roll back [root@localhost data] # cd / data/ [root@localhost data] # tar-czf fastdfs-1.49.bak.tar.gz fastdfs/# backup environment B data when the merge fails, and to migrate and merge [root@localhost data] # cd / data/ [root@localhost data] # tar-czf fastdfs-73.129.bak.tar.gz fastdfs/ 4. Migration merge # modify 192.168.1.49 / etc/fdfs/storage_ids.conf Increase information by 10002 (previously 10002 has been preconfigured as environment B) [root@localhost data] # cat / etc/fdfs/storage_ids.conf# 100001 group1 192.168.1.49 100002 group1 192.168.1.5 install FastDFS at 192.168.1.50 Modify the server information in / etc/fdfs/storage.conf (use the same tracker as 49 Bind_addr=192.168.1.50tracker_server=192.168.1.49:22122# restarts the tracker service [root@localhost data] # ps-ef | grep tracroot 1938 10 10:55 / usr/local/FastDFS/bin/fdfs_trackerd / etc/fdfs/tracker.confroot 2079 2014 0 11:49 pts/2 00:00:00 grep trac [root@localhost data] # kill 1938 [root@localhost Data] # / usr/local/FastDFS/bin/fdfs_trackerd / etc/fdfs/tracker.conf# copy the data of 172.18.73.129 to the 1.50s storage # start the 1.50s storage service [root@localhost etc] # / usr/local/FastDFS/bin/fdfs_storaged / etc/fdfs/storage.conf V, The binlog status of 1.49max 1.50 was checked respectively. You can see the log that the two nodes synchronize with each other, and the final state of the two nodes is ACTIVE and the synchronization is completed.

Use the md5 generated in step 2 to verify all the data on both nodes at the same time. Verify two md5 files on 1.49

Verify two md5 files on 1.50

You can see that only the MD5 check of the dat and binlog files fails, because the two files will be modified when they are merged, so don't worry here. In addition, there is no loss of files.

Seventh, access data verification. The url of environment An accesses the data of the original environment B and can be read normally.

At this point, the merger has been completed. Expand the node and verify all the data on the new node.

Previously, the data merge of the two environments has been completed. From the verification results, we can see that all user files have been synchronized normally. Here you can test and add another storage node to further demonstrate the conclusion of the successful merge. If the merged data can be completely synchronized to the new node, it means that the merged information is fully recorded in FastDFS to ensure that the merger is successful.

Add a new storage node, assuming IP:192.168.1.51, and specify an ID of 100003.

# modify 192.168.1.49 / etc/fdfs/storage_ids.conf to add 10002 information (previously 10002 has been preconfigured as environment B) [root@localhost data] # cat / etc/fdfs/storage_ids.conf# 100001 group1 192.168.1.49 100002 group1 192.168.1.50100003 group1 192.168.1.51 install FastDFS on 192.168.1.51 Modify the server information in / etc/fdfs/storage.conf bind_addr=192.168.1.51tracker_server=192.168.1.49:22122# restart tracker service [root@localhost ~] # ps-ef | grep tracroot 2082 10 11:49? 00:00:00 / usr/local/FastDFS/bin/fdfs_trackerd / etc/fdfs/tracker.confroot 2178 2014 0 12:14 pts/2 00:00:00 grep tra [ Root@localhost ~] # kill 2082 [root@localhost ~] # / usr/local/FastDFS/bin/fdfs_trackerd / etc/fdfs/tracker.conf# starts the storage service [root@localhost etc] # / usr/local/FastDFS/bin/fdfs_storaged / etc/fdfs/storage.conf checks storage information on 1.49

Verify the MD5 summary information on 1.51. At this point, you can see that the newly expanded node can synchronize all files in the two environments.

Upload a file on 1.49, and then check whether the file can be synchronized on three nodes. You can see that the newly uploaded files can be synchronized to all data nodes.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.