Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

High availability and high performance distributed file system FastDFS advanced keepalived+nginx for high performance on multi-tracker

2025-01-19 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >

Share

Shulou(Shulou.com)06/03 Report--

In the previous article on how to achieve high availability of distributed file system FastDFS, we have introduced the principle of FastDFS and how to build a simple high-availability distributed file system and how to access it.

High availability is achieved, but since we have only set up one group, if there are five servers now, there will be five servers with only one group, and the contents of the files in each server are the same (mutual backup) as shown in the following figure, which will result in a waste of resources.

Therefore, the following five servers are optimized and modified to further add keepalived+nginx multi-tracker architecture to achieve real high availability and high performance.

FastDFS cluster server distribution

Keepalived+nginx is used as a hot backup for 30 and 31 tracker servers. Use virtual VIP 27 as the unified entrance.

We will divide the servers according to the above architecture:

192.168.80.27 for virtual VIP

Keepalived+nginx hot backup, load balancing: 28,29

Tracker:

30 and 31 servers

Storage:

Group1:32 、 33

Group2:34 、 35

Although there is a Nginx on every server, let's emphasize the purpose of the Nginx here:

First of all, all Nginx are used for file access after upload, and have nothing to do with upload.

Above 28 and 29 are the common port loads, here is the load on two tracker access file ports 8000 (set by yourself).

The nginx installed on tracker is mainly to provide reverse proxy, load balancing and caching services for http access.

Nginx and FastDFS expansion modules are deployed on each storage server host. The Nginx module provides http download services for files stored in storage, and initiates redirect or proxy actions to the source storage host only when the current storage node cannot find the file.

Note: the tracker in the figure may be a cluster composed of multiple tracker, and the current Nginx expansion module of FastDFS supports multiple group on a single machine.

Tracker tracking nodes 30, 31

The installation and configuration steps are the same as before, configure tracker.conf and client.conf, and mainly pay attention to the firewall opening port 22122

Remember to manually create two folders:

/ data/fastdfs/tracker

/ data/fastdfs/client

The contents of both nodes are the same.

Storage storage nodes 32 to 35

Remember to manually create two folders: / data/fastdfs/storage

The installation steps are the same as above, and configure:

/ etc/fdfs/storage.conf

Group_name=group1 # group name (the first group is group1, the second group is group2, and so on.) base_path=/data/fastdfs/storage # data and log file storage root directory store_path0=/data/fastdfs/storage # the first storage directory, the second storage directory is named: store_path2=xxx Other storage directory names and so on. Tracker_server=192.168.80.30:22122# tracker server IP and port tracker_server=192.168.80.31:22122# tracker server IP and port

Note:

32 and 33 group_name=group1

34 and 35 group_name=group2

Other configurations are the same as the default, note that the firewall opens port 23000.

Open the two tracker and add boot:

/ usr/bin/fdfs_trackerd / etc/fdfs/tracker.conf

Chkconfig fdfs_trakcerd on

Open 4 storage and add boot:

/ usr/bin/fdfs_storaged / etc/fdfs/storage.conf

Chkconfig fdfs_storaged on

To view the status of the cluster, the overall content is displayed:

/ usr/bin/fdfs_monitor / etc/fdfs/storage.conf

Test service

/ usr/bin/fdfs_test / etc/fdfs/client.conf upload / etc/fdfs/anti-steal.jpg

The above is the installation and configuration of FastDFS. The following is introduced in combination with the Nginx module.

Upload nginx and fastdfs-nginx-module to storage storage nodes 32 to 35

The role of fastdfs-nginx-module has been described in the previous article, and here it is mainly configuration.

Upload fastdfs-nginx-module.tar.gz to / opt folder

Extract the plug-in package

Unzip fastdfs-nginx-module.zip

Nginx compilation and installation

Cd nginx

Re-config nginx

. / configure\-- prefix=/opt/nginx\-- pid-path=/opt/nginx/nginx.pid\-- lock-path=/opt/nginx.lock\-- error-log-path=/opt/nginx/log/error.log\-- http-log-path=/opt/nginx/log/access.log\-- with-http_gzip_static_module\-- http-client-body-temp-path=/opt/nginx/client\-- http-proxy-temp-path=/opt/nginx/proxy\ -http-fastcgi-temp-path=/opt/nginx/fastcgi\-- http-uwsgi-temp-path=/opt/nginx/uwsgi\-- http-scgi-temp-path=/opt/nginx/scgi\-- add-module=/opt/fastdfs-nginx-module/src

Then make & & make install

Copy mod_fastdfs.conf to fdfs for configuration

Cd fastdfs-nginx-module/src

Cp mod_fastdfs.conf / etc/fdfs/ # copy mod_fastdfs.conf to fdfs for configuration

Vim / etc/fdfs/mod_fastdfs.conf

Base_path=/data/fastdfs/storagetracker_server=.:tracker_server=.:storage_server_port=url_have_group_name=store_path0=/data/fastdfs/storagegroup_count= [group1] group_name=group1storage_server_port=store_path_count=store_path0=/data/fastdfs/ storage[group2] group_name=group2storage_server_port=store_path_count=store_path0=/data/fastdfs/storage

Note:

32 and 33 group_name=group1

34 and 35 group_name=group2

Configure nginx

Vim / opt/nginx/conf/nginx.conf

Add a Server:server {listen 8888; server_name localhost; location ~ / group ([0-9]) / M00 / {ngx_fastdfs_module;}} to the configuration file of nginx

8888 of the requests and those with group are transferred to the ngx_fastdfs_module plug-in for processing, not to mention the firewall.

Start Nginx and add and self-start

Shell > / opt/nginx/sbin/nginx

The files uploaded during the browser access test should be accessible normally.

Http://192.168.80.32:8888/group1/M00/00/00/CgNXGVpddOKANpzbAABdrZgsqUU369_big.jpg

Http://192.168.80.34:8888/group2/M00/00/00/CgNXGVpddOKANpzbAABdrZgsqUU369_big.jpg

Tracking node installation of Nginx and ngx_cache_purge modules

Description: each node performs the same operation

Tracker Node: 300.31

The nginx installed on tracker is mainly to provide reverse proxy, load balancing and caching services for http access.

Also upload nginx and ngx_cache_purge packages

Extract the file to the / opt folder

Tar-zxvf ngx_cache_purge-2.3.tar.gz

Then compile and install cd nginx. / configure-- prefix=/opt/nginx-- sbin-path=/usr/bin/nginx-- add-module=/opt/ngx_cache_purge-2.3make & & make install configure Nginx, set up tracker load balancing and cache

Vi / opt/nginx/conf/nginx.conf

Add the following

# set cache server_names_hash_bucket_size 128; client_header_buffer_size 32k; large_client_header_buffers 4 32k; client_max_body_size 300m; proxy_redirect off; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 90; proxy_send_timeout 90 Proxy_read_timeout 90; proxy_buffer_size 16k; proxy_buffers 464k; proxy_busy_buffers_size 128k; proxy_temp_file_write_size 128k; # set cache storage path, storage mode, allocated memory size, maximum disk space, cache duration proxy_cache_path / data/fastdfs/cache/nginx/proxy_cache keys_zone=http-cache:100m # set group1's server upstream fdfs_group1 {server 192.168.80.32 weight=1 max_fails=2 fail_timeout=30s;server 8888 weight=1 max_fails=2 fail_timeout=30s;server 192.168.80.33 weight=1 max_fails=2 fail_timeout=30s; 8888} # set group2's server upstream fdfs_group2 {server 192.168.80.34 listen 8888 weight=1 max_fails=2 fail_timeout=30s;server 192.168.80.35 listen 8888 server} server {listen 8000; server_name localhost # charset koi8-r; # access_log logs/host.access.log main; # set the load balance parameter location / group1/M00 {proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_cache http-cache; proxy_cache_valid 200 304 12h for group; proxy_cache_key $uri$is_args$args Proxy_pass http://fdfs_group1; expires 30d;} location / group2/M00 {proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_cache http-cache; proxy_cache_valid 200 304 12h; proxy_cache_key $uri$is_args$args Proxy_pass http://fdfs_group2; expires 30d;} # set the access permission to clear the cache location ~ / purge (/. *) {allow 127.0.0.1; allow 192.168.80.0amp 24; deny all; proxy_cache_purge http-cache $1 $is_args$args } # error_page 404 / 404.html; # redirect server error pages to the static page / 50x.html # error_page 500502503504 / 50x.html; location = / 50x.html {root html;}}

The above configuration is to select the response to the corresponding server according to group1 or group2 when requesting port 8000.

Note that the color has been marked. Also create a cache directory in advance: / data/fastdfs/cache/nginx/proxy_cache

Start Nginx/opt/nginx/sbin/nginx Settings Boot Boot: vi / etc/rc.local## join the following configuration / opt/nginx/sbin/nginxchmod + x / etc/rc.local# centos7 file access test

Previously, you accessed the file directly by accessing Nginx in the Storage node:

Http://192.168.80.32:8888/group1/M00/00/00/CgNXGVpddOKANpzbAABdrZgsqUU369_big.jpg

Http://192.168.80.34:8888/group2/M00/00/00/CgNXGVpddOKANpzbAABdrZgsqUU369_big.jpg

You can now access it through Nginx in Tracker:

(1) access through Nginx in Tracker1

Http://192.168.80.30:8000/group1/M00/00/00/CgNXGVpddOKANpzbAABdrZgsqUU369_big.jpg

Http://192.168.80.30:8000/group2/M00/00/00/CgNXGVpddOKANpzbAABdrZgsqUU369_big.jpg

(2) access through Nginx in Tracker2

Http://192.168.80.31:8000/group1/M00/00/00/CgNXGVpddOKANpzbAABdrZgsqUU369_big.jpg

Http://192.168.80.31:8000/group2/M00/00/00/CgNXGVpddOKANpzbAABdrZgsqUU369_big.jpg

The Nginx in each Tracker does load balancing to the back-end Storage group separately, but for the whole FastDFS cluster, if you want to provide a unified file access address, you also need to HA the Nginx in the two Tracker.

Configure Tracker server high availability, reverse proxy and load balancing

In fact, the above can access files through port 8000 of 30 or 31, and unify the port below to load balance 30 and 31 again.

A highly available load balancing cluster composed of Keepalived and Nginx is used to balance the load of Nginx in the two Tracker nodes.

Nginx can also be replaced with Haproxy here. The installation and introduction of Keepalived and Haproxy are described in detail in the spare tire formation record KeepAlived to achieve hot standby load and HAProxy load balancing of the pit series.

Using Nginx, install Keepalived and Nginx on 28 and 29, with a VIP of 27.

Vi / opt/nginx/conf/nginx.confhttp {include mime.types;default_type application/octet-stream;sendfile on;keepalive_timeout 65 witch # FastDFS Tracker Proxyupstream fastdfs_tracker {server 192.168.80.30 FastDFS Tracker Proxyupstream fastdfs_tracker 8000 weight=1 max_fails=2 fail_timeout=30s;server 192.168.80.31 Vera 8000 weight=1 max_fails=2 fail_timeout=30s;} server {listen 80th serverSecretname localhost;location / {root html;index index.html index.htm;} error_page 500502503504 error_page 500502503504ash location = / 50x.html {root html } # # FastDFS Proxylocation / dfs {root html;index index.html index.htm;proxy_pass proxy_set_header Host $http_host;proxy_set_header Cookie $http_cookie;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header X-Forwarded-Proto $scheme;client_max_body_size 300m } Boot Nginx/opt/nginx/sbin/nginx Settings Boot Boot: vi / etc/rc.local## join the following configuration / opt/nginx/sbin/nginxchmod + x / etc/rc.local# centos7 access files through VIP

Http://192.168.80.27:8000/dfs/group1/M00/00/00/CgNXGVpddOKANpzbAABdrZgsqUU369_big.jpg

Http://192.168.80.27:8000/dfs/group2/M00/00/00/CgNXGVpddOKANpzbAABdrZgsqUU369_big.jpg

The above is the whole configuration process, it looks very complicated, so be sure to figure out the entire deployment architecture first, and then know what each plug-in is used to do, and then have a clear idea if you want to expand.

Environment building is just the beginning, real use and optimization is the key point, so watch patiently.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 234

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Internet Technology

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report