Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what is the website hardware configuration that supports StackOverflow operation". In the daily operation, I believe that many people have doubts about what website hardware configuration supports StackOverflow operation. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "what website hardware configuration to support StackOverflow operation"! Next, please follow the editor to study!

The question-and-answer community network StackExchange consists of more than 100 websites, including StackOverflow, which is ranked 54th by Alexa. StackExchang has 4 million users and 560 million PV per month, but only 25 servers are used, and the CPU load is not high.

It does not use cloud computing because it may slow down, make it harder to optimize and troubleshoot the system.

StackOverflow still uses Microsoft's architecture, which is very practical. Microsoft's infrastructure works effectively and is cheap enough that there is no compelling reason to change. But that doesn't mean it doesn't use Linux, it uses Linux in a meaningful way.

Its Windows server runs an operating system version of Windows 2012 R2 that runs Centos 6.4.

The website database MS SQL size 2TB, all stored on SSD, has 11 Web servers running IIS, 2 load balancing servers running HAProxy, and 2 caching servers running Redis.

StackOverflow is an IT technical Q & A site where users can submit and answer questions. StackOverflow now has 4 million users, 40 million responses, and PV5.6 billion a month, ranking 54th in the world. However, it is worth noting that there are only 25 servers supporting their site, and all of them maintain a very low resource utilization rate, which is a contest in high efficiency, load balancing, caching, database, search and efficient code. Recently, Todd Hoff, founder of High Scalability, summed up the reasons for StackOverflow's success based on Marco Cecconi's speech video "The architecture of StackOverflow" and Nick Craver's blog post "What it takes to run StackOverflow".

Unsurprisingly, Stack Overflow still uses Microsoft products heavily. They argue that since Microsoft's infrastructure can meet demand and is cheap enough, there is little reason to make fundamental changes. Where needed, they also use Linux. Fundamentally, it's all about performance.

Another area of concern is that Stack Overflow still uses a scale-up strategy instead of the cloud. They use 384GB's memory and 2TB's SSD to support SQL Servers, and if they use AWS, the cost can be imagined. Another reason for not using the cloud is that Stack Overflow believes that the cloud will degrade performance to some extent and make it more difficult to optimize and troubleshoot the system. In addition, their architecture does not require scale-out. The peak period is a scale-out killer application scenario, but they have a wealth of system adjustment experience to deal with. The company still adheres to Jeff Atwood's famous saying that hardware is always cheaper than programmers.

Marco Ceccon has mentioned that when talking about systems, there is one thing that must be understood first-- the type of problem that needs to be solved. First, from a simple point of view, what exactly is StackExchange used for-first some topics, then build communities around those topics, and finally form this admirable question-and-answer site.

The second is scale correlation. StackExchange is growing rapidly and needs to deal with a large number of data transfers, so how do you accomplish all this, especially with only 25 servers? let's get to the bottom of it:

Status

StackExchange has 110 sites, growing at a rate of three to four a month.

4 million users

8 million question

40 million answers

Ranked 54th in the world

Annual growth rate of 100%

Monthly PV 560 million

Most weekdays peak at 2600 to 3000 requests per second, and as a programming-related website, requests on weekdays are generally higher than on weekends

25 servers

The SQL data of 2TB is stored in SSD

Each web server is configured with two 320G SSD, using RAID 1

Each ElasticSearch host is equipped with 300GB's mechanical hard drive, but also uses SSD

The read-write ratio of Stack Overflow is 40:60.

The average CPU utilization of DB Server is 10%

11 web server, using IIS

2 load balancers, 1 active, using HAProxy

4 active database nodes, using MS SQL

3 application servers that implement tag engine, and all searches are done through tag

3 servers do search through ElasticSearch

2 servers using Redis to support distributed caching and messages

2 Networks (Nexus 5596 + Fabric Extenders)

2 Cisco 5525MuthX ASAs

2 Cisco 3945 Routers

2 read-only SQL Servers for the main service Stack Exchange API

VM is used for deployment, domain controller, monitoring, operation and maintenance database, etc.

Platform

ElasticSearch

Redis

HAProxy

MS SQL

Opserver

TeamCity

Jil--Fast .NET JSON Serializer, based on Sigil

Dapper-- miniature ORM

UI

UI has a message inbox for getting new badges, sending messages from users, and collecting messages when major events occur, implemented in WebSockets and supported by Redis.

The search box is implemented through ElasticSearch and uses a REST interface.

Because users ask questions frequently, it is difficult to display the latest problems, and new problems arise every second, so it is necessary to develop an algorithm that focuses on user behavior patterns and only shows users the questions they are interested in. It uses complex Tag-based queries, which is why a stand-alone Tag Engine is developed.

Server-side templates are used to generate pages.

Server

25 servers are not fully loaded, CPU utilization is not high, and only 5 servers are needed to calculate SO (Stack Overflow).

The resource utilization of the database server is about 10%, when performing a backup.

Why is it so low? Because the database server has enough 384GB memory, and the CPU utilization of web server is only 10% 15%.

Vertical expansion has not encountered a bottleneck. Typically, about 100 to 300 servers are required for such traffic to scale out.

A simple system. Based on .net, only 9 projects are used, and 100 may be needed for other systems. The reason for using so few systems is to pursue the ultimate compilation speed, which needs to be planned from the beginning of the system, with a compilation time of about 10 seconds per server.

110,000 lines of code, very little compared to traffic.

This minimalist approach is used for several reasons. First of all, you don't need much testing, because Meta.stackoverflow is already a problem and bug discussion community. Second, Meta.stackoverflow is also a software testing site, if users find a problem, they will often put forward and give a solution.

The New York data center uses Windows 2012, which has been upgraded to 2012 R2 (Oregon has completed the upgrade), and the Linux system uses Centos 6.4.

SSD

Intel 330 (Web layer, etc.) is used by default

Intel 520 for middle-tier writes, such as Elastic Search

The data layer uses Intel 710 and S3700

The system uses both RAID 1 and RAID 10 (any disk above 4 + uses RAID 10). Don't be afraid of failure, even if thousands of 2.5in SSD are used in the production environment, you haven't encountered a single failure situation. Each model uses more than one spare, and the scenario of multiple disk failures is not under consideration.

ElasticSearch performs extremely well on SSD because SO writes/re-indexes operates very frequently.

SSD has changed the way search is used. Because of locking problems, Luncene.net cannot support the concurrent load of SO, so they turn to ElasticSearch. In an all-SSD environment, there is no need to build locks around the Binary Reader.

High availability

Remote backup-the primary data center is located in New York and the backup data center is in Oregon.

Redis has two slave nodes, SQL has two backups, Tag Engine has three nodes, and elastic has three nodes, redundant everything, and exists in both data centers.

Nginx is used for SSL, and the transformation uses HAProxy when terminating SSL.

Not all master and slave, some temporary data will only be put in the cache

All HTTP traffic sends only 77% of the total traffic, and there are backups of Oregon data centers and some other VPN traffic. This traffic is mainly generated by SQL and Redis backups.

Database

MS SQL Server

Stack Exchange has a database set up for each site, so Stack Overflow has one, Server Fault has one, and so on.

In the primary data center in New York, each cluster usually uses a configuration of 1 primary and 1 read-only backup, and a backup is also set up in the Oregon data center. If you are running an Oregon cluster, both backups in the New York data center will be read-only and synchronized.

A database prepared for other content. There is also a "network-wide" database for storing login credentials and aggregate data (mostly stackexchange.com user files or API).

Careers Stack Overflow, stackexchange.com, and Area 51 all have their own independent database schemas.

Schema changes need to be made available to databases of all sites at the same time, and they need to be backward compatible. For example, if you need to rename a column, it will be very troublesome, and you need to do multiple operations: add a new column, add code that works on two columns, write data to the new column, change the code to make the new column valid, and remove the old column.

There is no need for sharding, everything is solved by indexing, and the data volume is not that large. If there are filtered indexes requirements, why not do it more efficiently? Common patterns are indexed only on DeletionDate = Null, while others are indexed by specifying types for enumerations. A table is set for each votes, such as a table for post votes,1 and a table for comment votes. Most pages can be rendered in real time and are cached only by anonymous users, so there are no cache updates, only re-queries.

Scores is non-normalized, so it needs to be queried frequently. It contains only IDs and dates,post votes tables, which currently have about 56454478 rows. Using the index, most queries can be completed in milliseconds.

Tag Engine is completely independent, which means that the core functionality does not depend on any external application. It is a large array of memory structures optimized for SO use cases and precomputed for heavy load combinations. Tag Engine is a simple windows service that runs redundant on multiple hosts. The utilization rate of CPU is basically maintained at 2-5%. Three hosts are dedicated to redundancy and are not responsible for any load. If all hosts fail at the same time, the network server will load the Tag Engine into memory and run continuously.

About the comparison between Dapper compiler-free check query and traditional ORM. There are many benefits to using a compiler, but there are still fundamental disconnect problems at run time. And more importantly, due to the generation of nasty SQL, it is usually necessary to find the original code, and the lack of capabilities such as Query Hint and parameterization control makes query optimization more complex.

Coding

Process flow

Most programmers work remotely and choose their own coding location.

Compiling is very fast

Then run a small number of tests

Once compiled, the code is transferred to the development delivery preparation server

Hide new features through function switches

Run as a test at another site on the same hardware

Then move to Meta.stackoverflow testing, which is used by thousands of programmers every day, a good testing environment.

If it passes, go online and test it in the wider community.

Heavy use of static classes and methods for simpler and better performance

The coding process is simple because the complex parts are packaged into libraries, which are open source and maintained. The number of .net projects is low because part of the code shared by the community is used.

Developers use two or three monitors at the same time, and multiple screens can significantly improve productivity.

Caching

Cache everything

5 levels of caching

Level 1 is a network-level cache that is cached in browsers, CDN, and proxy servers.

Level 2 is accomplished by the .net framework HttpRuntime.Cache, in the memory of each server.

Level 3 Redis, distributed memory key storage, sharing cache items on multiple servers supporting the same site.

Level 4 SQL Server Cache, the entire database, all the data is put in memory.

Level 5 SSD. It usually takes effect only after the SQL Server is preheated.

For example, each help page is cached, and the code to access a page is very simple:

Static methods and classes are used. It's really bad from an OOP perspective, but it's very fast and good for concise coding.

The cache is supported by Redis and Dapper, a mini ORM

To solve the garbage collection problem, one class in the template uses only one copy, which is created and saved in the cache. Monitor everything, including GC fuck. According to statistics, when the indirect layer increases the GC pressure to a certain extent, it will significantly reduce the performance.

CDN Hit . Since the query string is hashed based on the contents of the file, it will be fetched again only if there is a new creation. 30 million to 50 million Hit per day, with a bandwidth of approximately 300GB to 600GB.

CDN is not used to deal with CPU or Icano load, but to help users get answers faster.

Deployment

Deploy 5 times a day without building oversized applications. Mainly because

Performance can be monitored directly

Minimize the establishment as much as possible, and work is the key.

After the product is established, it is copied to each web page layer through a powerful script. The steps for each server are as follows:

Notify HAProxy to remove a server through POST

Delay IIS to end an existing request (approximately 5 seconds)

Stop the website (end all downstream through the same PSSession)

Robocopy file

Open the website

Do HAProxy Re-enable through another POST

Almost all deployments are over puppet or DSC, and upgrades are usually just about drastic adjustments to the RAID array and installation via PXE boot, which is very fast.

Collaboration

party

SRE (System Reliability Engineering): 5 people

Core Dev (Quan A site) 6-7 people

Core Dev Mobile:6 people

Careers team is responsible for SO Careers product development: 7 people

Devops and developers are very closely integrated.

There are great changes between teams.

Most employees work remotely

The office is mainly for sales, with the exception of Denver and London

Everything is equal and somewhat biased towards New York workers, because face-to-face is conducive to work communication, but online work does not have much impact.

Compared to being able to work in the same office, they prefer product-loving and talented engineers, who can weigh the pros and cons very well.

Many people choose to work remotely because of their families. New York is good, but life is not easy.

The office is in Manhattan, which is the birthplace of talent. The data center should not be too biased, as upgrades are often involved

Build a strong team with a preference for geeks. Early Microsoft gathered a large number of geeks, so they conquered the whole world.

The Stack Overflow community is also a recruiting place, where they are looking for people who love coding, helpful and communicative.

Prepare a budget

Budget is the foundation of the project. The money is only spent on building infrastructure for new projects, and web server, which is so underutilized, was bought when the data center was established three years ago.

test

Rapid iteration and abandonment

Many tests are done by the release team. The developer has the same SQL server and runs on the same Web tier, so the performance tests are not bad.

Very few tests. Stack Overflow doesn't do much unit testing because they use a lot of static code and a very active community.

Infrastructure change. Since everything has two copies, every old configuration has a backup and uses a quick failure recovery mechanism. For example, keepalived can roll back quickly in a load balancer.

They prefer to rely on redundant systems rather than regular maintenance. SQL backups are tested with a dedicated server just so that they can be restored. Plan to do full data center failure recovery every two months, or use a completely read-only second data center.

Unit testing and integrated test box UI testing are done for each new feature release, which means that the product input can be predicted and then pushed to the incubator website, namely meta.stackexchange (former meta.stackoverflow).

Monitoring / Log

Http://logstash.net/ is currently being considered for log management, and a special service is currently used to transfer syslog UDP to the SQL database. Header is added to the web page for timing so that it can be captured through HAProxy and integrated into the syslog transmission.

Opserver and Realog are used to display measurement results. Realog is a log display system built by Kyle Brandt and Matt Jibson using Go.

Logging is done through the HAProxy load balancer with the help of syslog, not IIS, because it is more functional than IIS.

About Cloud

It's still a clich é that hardware is always cheaper than developers and efficient code. Based on the bucket effect, the speed must be limited by a certain deficiency, and the existing cloud services basically have capacity and performance limitations.

If we had used the cloud to build SO from the beginning, we might have reached the current level. But there is no doubt that if the same performance is achieved, the cost of using the cloud will be much higher than that of a self-built data center.

Performance is paramount

StackOverflow is a heavy performance control. The loading time of the home page is always limited to 50 milliseconds, and the current response time is 28 milliseconds.

Programmers are keen to reduce page load time and improve the user experience.

Each separate network submission is timed and recorded, and this measurement can identify areas that need to be modified to improve performance.

The main reason for such low resource utilization is efficient code. The average CPU utilization of web server is between 5 and 15 per cent, memory usage is 15.5 GB, and network traffic is between 20 Mb/s and 40 Mb/s. The CPU usage of the SQL server is between 5 and 10 percent, the memory usage is 365GB, and the network transport is from 100 Mb/s to 200 Mb/s. This can bring three benefits: leave a lot of room for the upgrade, keep the service available when a serious error occurs, and roll back quickly when needed. I prefer to think of Stack Overflow as being able to run on large-scale data, but not running with scale but not at scale in itself. It means that our website is very efficient, but at least so far, our scale is not big enough. Let's use some numbers to show what the current size of Stack Overflow is. Here are some core figures from statistics made not long ago in a whole day (24 hours), November 12, 2013 to be exact. This is a typical working day and only counts our active data centers, that is, our own servers. Requests and traffic to CDN nodes are excluded because they do not directly access our network.

The load balancer accepted 148084833 HTTP requests

Of these, 36095312 were loaded pages.

833992982627 bytes (776 GB) HTTP traffic for sending

A total of 286574644032 bytes (267GB) data were received

A total of 1125992557312 bytes (1048 GB) data were sent

334572103 SQL queries (only from HTTP requests)

412865051 Redis requests

3603418 tag engine requests

Spend 558224585 ms (155hours) on SQL query

Spent 99346916 ms (27 hours) on Redis requests

Spent 132384059 ms (36 hours) on tag engine requests

It takes 2728177045 ms (757 hours) on ASP.Net program processing.

(I think we should publish an article on how we collect these data quickly and why it is worth the effort to obtain them.)

Note that the above figures include the entire Stack Exchange network, but that's not all of us. In addition, these numbers come only from the HTTP requests we recorded to test performance. "Wow, how do you do that with so many hours in a day?" We call this magic, and of course some people like to say "many servers with multi-core processors", but we still insist that it is magic. Here are the devices running the Stack Exchange network in that data center:

4 MS SQL servers

11 IIS servers

2 Redis servers

3 tag engine services (any request for tags will access them, such as / questions/tagged/c++)

3 ElasticSearch servers

2 load balancers (HAProxy)

2 switches (Nexus 5596 and Fabric Extenders)

2 x Cisco 5525Murx ASA (can be thought of as a firewall)

2 x Cisco 3945 Router

There is a picture and the truth:

We not only run the website, but also have some servers and other devices running virtual machines on the next shelf, which do not serve the website directly, but carry out deployment, domain name control, monitoring, database operation and other work. The two database servers in the above list have been used as backups until recently as read-only payloads (mainly for Stack Exchange API), so we can continue to scale up without much thought. There are two Web servers for developing and storing metadata, and the running load is very low.

Let's sum up again:

Core equipment

If you get rid of those extra devices, the following are required for Stack Exchange to run (maintain the current level of performance):

2 MS SQL servers (the Stack Overflow is on one, the other is on the other, and there is actually a surplus if only one machine is running)

2 Web servers (maybe 3, but I'm confident 2 will be enough)

1 Redis server

1 label engine server

1 ElasticSearch server

1 load balancer

1 switch

1 ASA

1 router

We should really find a chance to try this configuration, turn off some of the devices and see what the limit is.

Now there are still some virtual machines running in the background, performing some auxiliary functions, such as domain name control and so on. But those are fairly low-load tasks, so let's not talk about it. Let's focus on Stack Overflow itself and see how it loads out of the page at full speed. If you want to be more accurate and comprehensive, you can add a VMware virtual machine to perform all the auxiliary work. It doesn't seem to require a lot of machines, but the specifications of these machines are usually difficult to achieve in the cloud unless you have enough money. The following is a brief introduction to the configuration of these "enhanced" servers:

The database server has SSD hard drives with 384GB memory and 1.8TB

Redis server has 96GB memory

ElasticSearch server has 196GB memory

The tag engine server has the fastest processor we can afford

Each port of the switch has the bandwidth of 10Gb

The Web server is not very special, with 32GB memory, two 4-core processors and 300GB SSD hard drives

Some servers have 2 10Gb bandwidth interfaces (such as databases), others have 4 1Gb bandwidth

Is the bandwidth of 20Gb too redundant? You're damn right, the average active database server uses only 100-200Mb in the 20Gb channel. However, operations such as backup, reconstruction, and so on, can completely saturate the bandwidth according to the current memory and SSD hard drives, so this design makes sense.

Storage Devic

We currently have about 2TB database storage (the first cluster has 18 SSD drives-a total of 1.63TB, using 1.06TB; the second cluster consists of 4 SSD drives-a total of 1.45TB, using 889GB), which is what we need on the CVM (mm-hmm, complaining about the price again), keep in mind that these are all SSD drives. Thanks to the good performance of the memory, the average write time of our database is 0 milliseconds, even more accurate than we can measure. Taking into account in-memory data and two levels of cache, the actual database read-write ratio in Stack Overflow is 40:60. You read it correctly, 60% are write operations (click here to learn about the read-write ratio). In addition, each Web server has a RAID1 consisting of two 320GB SSD hard drives. ElasticSearch requires about the capacity of 300GB in each block, and because we write or re-index very frequently, SSD hard drives are a better choice here.

It is worth noting that we have a SAN (storage area network) connected to the core network, which is Equal Logic PS6110X, which has 24 hot-swappable 10K SAS disks and 2 10Gb controllers. This device is only used by the VM server as a shared storage space to ensure the high availability of the virtual machine, but does not actually support the operation of the website. In other words, if the SAN goes down, the site won't even notice it for a while (only the domain name controller in the virtual machine can sense it).

Integrate together

What is all this equipment for? Performance. We need high performance, which is a very important feature for us. The home page of all sites is a question page, which we affectionately call Question/Show (the name of the route). On November 12th, the average rendering time for this page was 28 milliseconds, while our requirement was at most 50ms. In order to give users a better experience, we do everything possible to shorten the page loading time, even if only for a millisecond. All of our developers are "penny-pinching" when it comes to performance-related issues, which also helps our website stay responsive. Here is the average rendering time for some of the most popular pages on Stack Overflow, and the data are still from the previous 24 hours:

Question/Show: 28 ms (29.7 million clicks)

User Profiles: 39 ms (1.7 million clicks)

Question List: 78 ms (1.1 million clicks)

Home page: 65 ms (1 million hits) (this is already slow for us, and Kevin Montrose is working to fix the problem)

With the recording of the timeline of each request, we can accurately observe the process of page loading. We need this kind of data, otherwise do we rely on brain supplement to make decisions? With data on hand, we can monitor performance like this:

If you are interested in the data on a particular page, I would be happy to publish it. But here I focus on rendering time, because it shows how long it takes our server to generate a web page. Network speed is a completely different topic (although I have to admit it has a lot to do with it), but I'll talk about it in the future.

Growth space

It is worth mentioning that the utilization of these servers is very low. For example, the average CPU utilization rate of the Web server is 5-15%, the memory is only 15.5GB, and the network traffic is only 20-40Mb., while the average CPU utilization rate of the database server is 5-10%, using 365GB memory and 100-200Mb/s network. This enables us to do several important things: when the size of the site increases, there is no need to upgrade the device immediately; when something goes wrong (wrong queries, code, attacks, etc.), we can keep the site open all the time; reduce power consumption when necessary. Here is a monitoring project for our web layer:

The main reason for such low utilization is efficient code. Although this is not the topic of this article, efficient code also plays a decisive role in mining server performance. Doing something unnecessary costs more than doing nothing-extending this to your code means you need to improve them more efficiently. These losses or consumption can be energy, hardware (you need more and faster servers), and it is more difficult for developers to understand code (to be fair, this has two sides, efficient code is not necessarily that simple). And slow page rendering-may cause users to visit other pages of the site less or even never visit your site again. The cost of inefficient code may be much greater than you think.

At this point, the study on "what is the hardware configuration of the website that supports the operation of StackOverflow" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.