Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "what are the main technologies developed by Quora". In the operation process of actual cases, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations! I hope you can read carefully and learn something!

First, let's look at the main technologies used in Quora website development:

Web Layer and CMS

HAProxy acts as a front-end Load Balancer server, and the reverse proxy server is Nginx, followed by Pylons(Pylons + Paste), which takes on dynamic Web requests.

Webnode2 and LiveNode are two internal systems responsible for creating and managing content. Webnode2 generates HTML, CSS and JavaScript and is lightly coupled to LiveNode. LiveNode's role is to display Web page content. Written in Python, C++ and JavaScript. Special mention is made of the use of jQuery and Cython. LiveNode may be open source.

Why Python?

Some Python related technical components have already been mentioned. It's interesting that the team that came out of Facebook actually used Python as their primary development language. Quora has an explanation for this:Facebook chose PHP not because it was the best choice, but for historical reasons. The Quora technical team chose Python after examining multiple languages, of course, for a number of reasons, and overall, it was not very radical.

communication processing

Backend communication uses Thrift, which is open source from Facebook. In addition to the simple development interface, it may be more familiar. It is also a factor:)Comet server uses Tornado to handle Long polling and Push updates (I don't know what to use?). Tornado is an open source product of the former FriendFeed technology team.

real-time search

Because Sphinx doesn't meet real-time requirements, Quora uses its own search engine, using only Thrift and Python Unicode libraries, and nothing else. Quora's search is special, because it needs to correlate input content and make effective prompts, so it needs to provide better prefix indexing.

The implementation of Quora search is quite technical, and the pressure on the backend query request is not small (perhaps the current concurrent request volume is not so large). For this scenario, friends who do relevant development may wish to study it carefully. If the general framework is similar, it is likely that those details will determine the final outcome.

data persistence layer

MySQL is heavily used as a storage scheme and Memcached as a Cache layer. No currently popular NoSQL related products are used. Quora has its own reasons for doing so. SNS sites that do not have millions of users do not need NoSQL at all. Maybe Quora will be enabled in the future.

Founder Charlie Cheever and Adam D'Angelo were both at Facebook, so Quora's technology does have a lot of Facebook genes. Quora's team size is not large, it is estimated that more than ten people do technology, such a compact team uses so many technologies and products, it can be seen that many people are versatile. This is where domestic technical teams need to learn from their foreign counterparts.

Quora's technical management experience:

Quora's four principles of code quality

1. Reading and understanding code should be simple-developers spend far more time reading code than writing it. Quora should therefore try to keep the code simple and understandable, although writing that code may take longer than it should.

2. Different parts of code should have different quality requirements-different parts of code have different long-term development impacts because they have different life cycles, impact areas, vulnerability, consequences, and debug difficulties. In general, different codes have different effects on the speed of iteration of the product, so treating all codes equally is obviously not the best choice.

3. The cost of maintaining code quality can be reduced-development automation, easier development tools, better processes, and better developers can all reduce the cost of maintaining code quality.

4. Consistency across the code base-Consistency across the code base is important, although it means that some parts of the code may not have been written in the best way. A code base that lacks consistency makes it harder to read, understand (see point 1), add features later, and use automated tools to improve the code.

Next, I'll describe some of the specific ways Quora follows the four principles during development.

Mutual review after code submission (Code Review)

If there is a code change in the code base, Quora peer-reviews it in six ways-correctness, code privacy, performance, architecture, reusability, and code style. Reading code is an indispensable part of code review, so implementing code review system is undoubtedly beneficial to increase code readability.

Unfortunately, code reviews can also slow down development. For example, it is an industry practice for code to be reviewed by peers and improved by authors before being submitted to the codebase. Each review round may last two days, and then there are often two or three rounds of reviews, which means that code authors often waste more than half a week on code reviews.

At Quora, Quora does not conduct pre-submission reviews. That is, the code goes live before it is reviewed by some colleagues. To give you an idea of the scale of Quora's work, Quora had 48 developers make a total of 187 commits yesterday. Quora thinks post-code review is a great move because it frees developers from the prison of unnecessary code reviews and allows them to finish other work first. This way, reviewers can also read the code at their convenience, avoiding being pushed to do the job. Quora's process expects Quora to complete its code review within a week, but most Quora reviews occur within one to two days. The length of this "week" was carefully discussed-long enough for reviewers to schedule their review time freely, but short enough to prevent low-quality code from being read and used by others. In practice, Quora also takes into account that many developers will have a work schedule scheduled on a "week" basis.

Quora is able to implement this code review after submission because Quora trusts every developer in Quora. After all, Quora only hires the best developers and gives them the best tools and the most careful training. This also prompted Quora to write good tests to achieve high code coverage, which allowed Quora to check code correctness before any code reviews. In addition, Quora uses Phabricator, a great code review tool with a high degree of freedom in configuration. Quora has made some changes to Phabricator to better serve Quora's post-submission review process. For example, Quora wrote a command-line tool to help people get their code online and ask for reviews. This tool allows developers to make Phabricator diff work correctly without making any changes to previous commits.

Having said that, Quora has different review requirements for different kinds of changes. If new code has serious consequences and is difficult to fix, Quora will require a pre-commit review instead of the regular post-commit review. For example:

1. code relating to user privacy and anonymity;

2. code related to a core abstract class is involved, because a lot of other code may be based on it;

3. involving underlying code that could cause downtime;

Whether review is required before or after submission also depends on the developer's caution. If any developer wants to ask for a code review before submitting code to get some advice or comment, they are completely free to do so-although this rarely happens.

Send code reviews to the right people

For code reviews to go smoothly, new code should be reviewed by someone who has sufficient knowledge of the change. It would be nice if the code was owned by people who would maintain it, and obviously they would have good reason and motivation to optimize the long-term usability of the code.

Quora writes a simple system that allows developers to easily attribute code at the module/directory level by adding a meta tag to the beginning of the file. For example:

__reviewer__ = ‘vanessa, kornel’

If a commit changes a file, the system reads the tag, and the developer identified in the tag is automatically added to the commit's reviewer list. Quora also has other rules for code reviewers, such as for all commits with A/B tests, a data analyst is automatically added to the review list if none exists. In fact, Quora's tools can easily add other rules about code reviewers as well. Quora, for example, could send all of a new employee's commits to their mentor if it wanted to.

There is an intelligent review distribution system that sends code reviews to the right people, reducing the hassle of finding code reviewers and ensuring that the right people are found to review each piece of code.

test

Testing is an important part of the development process. Quora writes a lot of unit tests, functional tests, UI tests to achieve high code coverage. There is a complete unit test that allows developers to quickly complete new code in parallel without worrying about breaking existing functionality. Quora spent a lot of time creating Quora's test framework (based on nosetests), with the goal of being simple, fast, and easy to use, and keeping the workload of writing tests as low as possible.

Quora has also developed a number of automated testing tools. As discussed in a previous article on Quora Continuous Deployment, all Quora code is tested on a central server before going live. The test server is highly parallel, so running all Quora tests takes less than 5 minutes. This fast system is designed to encourage developers to write and run as many tests as possible. Quora has a tool called test-local to automatically find and execute tests related to new code. In order to use the tool well, Quora tests must be modular (this also helps developers find bugs quickly and fix them if a test fails). To ensure this and other important properties of the test code, Quora has a shared document describing the test code writing specifications. These principles are rigorously enforced during code reviews.

Similar to code reviews, Quora has different test criteria for different kinds of changes. Quora will require higher code coverage if new changes are likely to have serious consequences and repair costs.

All of this adds up to maximizing the value of Quora writing tests, thereby reducing long-term development costs.

Code Quality Guidance

Quora is very keen on sharing code quality guidance, which helps Quora:

1. Better training for new developers;

2. Better share Quora's experience and wisdom throughout the team;

3. Establish common standards to improve code base consistency;

4. Reduce workload in development and review. For example, discuss in each review whether 80 or 100 characters per line of code is meaningless. Quora can discuss this issue once and then use this standard in all future code.

In addition to each language's own statement specifications, Quora also has more abstract code specifications, such as how to write good tests or how to structure code modules to help reduce code reading time. These norms are not static. As Quora develops a deeper understanding of trade-offs, Quora changes these norms to maximize benefits. Quora also has large code refactoring tools (some open source tools such as codemod, others developed by Quora itself) to go back and refactor all the old code after Quora changes a specification.

Clean up the old code

A fast-moving team will try many new things, and naturally some of them will be good and some will be bad. Therefore, a fast-moving company's code base is bound to have a lot of dross, code that no one actually uses anymore but stays there to complicate a lot of things. Keeping the code base clean by being clear of these dross will also speed up development.

Quora regularly organizes "cleanup weeks" to clean up these dross. During these clean-up weeks, designated teams--and sometimes entire companies--spend all their time cleaning up old code that isn't used. These regular activities reduce the time and effort required to switch between "regular work" and "clean up work," and make cleaning up old code more fun and socially valuable.

Some parts of the code are easier to clean up than others, but there are also parts of the code that can have a huge impact on development speed. To make the best use of the time people spend cleaning up old code, Quora prioritizes the cleaning up of code modules based on how long it takes to clean them up and how quickly they affect development.

Code error checking and optimization (Linting)

It's easy to underestimate the consequences of occasionally failing to follow code language specifications (such as code comments or length of lines), but these consequences add up. Indeed, it can be annoying to remember and follow a lot of rules all the time, especially when there are more and more rules. So Quora isn't surprised that many developers aren't prepared to follow these specifications.

Quora has developed a company-wide error-checking and optimization tool called qlint to reduce the effort involved in achieving this goal. Qlint is a smart gadget based on flake8 and pylint that recognizes text structure and abstract syntax trees (AST). This tool makes it easy for Quora to add new code rules in the future. For example, Quora specifies that all private variables in Python must be underlined before the variable name, so Quora adds this rule to qlint to find all code that does not conform to this specification.

Quora integrates qlint into many other development tools, so developers don't have to pay special attention to the problems qlint points out. For budding developers, Quora integrates qlint into some of the most popular editors, such as Vim, Emacs, and Sublime, and provides visual cues when developers violate code specifications. Qlint is also integrated into the submission process and runs in an interactive way whenever anyone wants to submit code. In fact, depending on which rule commit violates, the tool may even prevent code deployment. Quora also incorporates Quora's code specification documentation into the tool, so for every rule a developer violates, qlint will give a link to that rule entry in the documentation. Quora's Phabricator is also configured to use qlint. This way, code review becomes easier because all errors are visually pointed out by qlint.

All of these improvements improve the consistency of Quora's codebase and enable Quora to improve code quality at minimal cost.

"What are the main technologies developed by Quora" is introduced here. Thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.