Shulou(Shulou.com)06/02 Report--

Background

Go-fastdfs is a distributed file system that supports http protocol. In general projects, the address of the file system is rarely exposed directly, and most of them are replaced by software such as nginx. Due to the relatively special business and network environment of our company, a hybrid cloud network system is composed of public network part (public cloud) and intranet part (private cloud). Public cloud is mainly used as an exit and entrance, as well as running some audit authentication and other applications to process upstream requests, so as to reduce the processing times of private cloud and improve performance. Then it is precisely because of this that in a public network environment, reverse proxies must be used to access services provided by private clouds. By the same token, the same is true for access to the file system. How can you configure it in nginx so that external network requests can be reversed to go-fastdfs? This article will elaborate step by step.

General configuration

In general, friends who are familiar with nginx know that if you need to configure a reverse proxy, you can write a location context and proxy module directly, and if you need a custom prefix, use a rewrite module. Simple examples are as follows:

Location ~ / dfs/group ([0-9]) {proxy_pass http://localhost:8080; rewrite ^ / dfs/ (. *) $/ $1 break; proxy_set_header Host $host:$server_port; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;}

At this point, the general reverse configuration is fine, so is it okay for go-fastdfs? For go-fastdfs upload is generally fine, but for the use of tus to do breakpoint continuation is still not enough, why? Because the tus server will return 301redirection and need to carry a certain request header, it also needs to be set up specially.

Reverse configuration that supports Tus

If you need to reverse tus, to support rewriting of 301redirect Location and to support a certain request header forwarding, how to configure it? Please refer to the configuration below

Location ~ / dfs1/group ([0-9]) {access_log logs/dfs/access.log main; error_log logs/dfs/error.log error; rewrite ^ / dfs1/ (. *) $/ $1 break; proxy_pass http://localhost:8051; # Disable request and response buffering proxy_request_buffering off; proxy_buffering off; proxy_http_version 1.1; proxy_set_header Host $host:$server_port; proxy_set_header X-Real-IP $remote_addr Proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # if server_name is not a public network domain name, this place can be set to ip proxy_set_header X-Forwarded-Host $hostname; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade" # because the prefix plus rewrite is used, so modify the returned Location plus the reverse proxy prefix proxy_redirect ~ ^ (. *) / group ([0-9]) / big/upload/ (. *) / dfs/group$2/big/upload/$3; client_max_body_size 0;}

Note that proxy_redirect and client_max_body_size are the two configurations above. The first configuration is because the redirect Location returned by the tus server does not carry a custom prefix, so you need to add a custom prefix. Here is / dfs. If it is something else, just change it. The second is client_max_body_size. Setting this to 0 means that no matter how big the file is uploaded, it will not be reported to request too large. Just forward it. If you need to set it, please set a number greater than or equal to chunkSize. What is chunkSize? This is the size of each part when uploading parts on the tus client. For details, please refer to the official documentation.

Load balancing configuration

When a cluster server is configured, how to load balance upload or download? The reverse generation is done with nginx, which can be realized with the upstream module. For more information, please refer to the configuration below.

Upstream dfs_stream {server host1:port; server host2:port; ip_hash;}

The above configuration is no different from the general load balancer. The only thing to pay attention to is to configure ip_hash. Why? Because when using breakpoints to resume upload, the files are uploaded in parts, and if it is not ip_hash, the first few pieces may be uploaded to A server, and the next few pieces will be uploaded to B server, so this kind of file is not complete, so you need to pay attention to this problem.

The above is the whole content of this article, I hope it will be helpful to your study, and I also hope that you will support it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.