Shulou(Shulou.com)06/03 Report--

This article introduces the relevant knowledge of "what is middleware development". In the operation process of actual cases, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations. I hope you can read carefully and learn something!

What is middleware development?

When I was a senior intern, I first came into contact with middleware in a software development team of more than 20 people. At that time, the project architect introduced Motan, an RPC framework based on Weibo open source. With this framework, we quickly built an internal e-commerce system based on microservice architecture. Then in the project, due to business requirements, we introduced ActiveMQ... At this stage, I was already using middleware, but I didn't seem to be touching middleware development, more at the usage level.

In my first job after graduation, the company had hundreds of R & D. The leader at that time saw that I was interested in middleware and intentionally assigned me to the infrastructure team. For the first time, I really became a "middleware R & D." Usually, my main work was based on open source Kong and Dubbo, and some internal transformation was carried out to provide better use for business teams. At this stage, the things to do are still relatively miscellaneous. The business team has customized requirements for some middleware, so they need to understand this middleware and be familiar with the source code. This period of time is also the fastest growing period for me. During this period, I learned Docker, Neo4j, Kong and other technologies that I had never touched before, and more deeply understood the principles of RPC frameworks such as Dubbo. Perhaps sitting next to me is a classmate in the order department who threw a function point for me to modify.

Now, I work for Aliyun native middleware. Compared with the previous middleware research and development work, Alibaba Cloud and other large Internet companies have a few people and dozens of people in charge of any middleware. There are also clear boundaries between the middleware department and the business department. Here, the middleware team's responsibilities can be broken down into three directions:

HarmonyOS Technology Community

The middleware team is driven by the needs of the business team and provides customized solutions within the group, commonly known as "self-development". So you may not know HSF, Diamond these Ali internal middleware.

Middleware teams will engage in open source, spending a lot of energy to improve the ultimate performance of middleware, enhance the impact of open source, and lead the technology. This part of the middleware is more well-known, such as Dubbo, Spring Cloud Alibaba, RocketMQ, Nacos.

Middleware will export commercial products in Alibaba Cloud, providing higher SLA, more powerful functions and friendlier interactions than open source products. Commercial products such as: Microservices Engine MSE, Message Queue RocketMQ, Distributed Application Link Tracing ARMS.

These three experiences reflect the different needs of middleware development for companies of different sizes. Small companies use middleware, such as RPC, MQ, caching, etc., which is largely maintained by business developers themselves. However, if the background R & D reaches hundreds of people, they will basically set up their own middleware team, or choose to use middleware products provided by cloud vendors such as Alibaba Cloud.

Difference between middleware development and business development

In my opinion, there is no difference between middleware development and business development. If you have to say the difference, it is a bit like the different classes in the game. Some people choose to be mages, while others choose to be warriors. In the process of leveling up in the workplace, everyone's total skill points are the same, business development is a bit towards the direction of all-round warriors, each point is involved in a point, but each direction can be assigned to the skill points naturally less; middleware development is like "because too afraid of pain, full point defense" in the protagonist, the skill points are assigned to a direction.

Suppose you are working in a small company, there is no dedicated middleware team at this stage, everyone is business development, at this time we make an assumption: the company is about to set up a middleware team or infrastructure department, then which type of person is easy to be selected? Must be those who have solid technical skills, interested in middleware, and have studied the source code. This assumption isn't made up, and many internet companies 'middleware teams grow up this way. What do I mean? There is no clear boundary between business development and middleware development at the beginning, so don't worry about whether you are engaged in business development now, as long as you are interested in middleware and have studied at the source level, you can become a middleware development.

What qualities do middleware developers need?

The larger the company, the larger the middleware team, the more vertical the division of responsibilities. Basically in large companies, a middleware development may take years to plow in a vertical direction. The following are some common middleware directions, although this classification may vary slightly from company to company due to organizational reasons.

HarmonyOS Technology Community

Microservices governance. Examples include RPC related middleware, registry, configuration center, current limiting fuse, link tracing, etc. Open source products such as Dubbo, SpringCloud, Nacos, Zookeeper, Sentinel, Hystrix, Zipkin.

Message queue. Microservices generally emphasize synchronous communication, and message queues are listed separately, mainly because of their asynchronous mechanism. Open source products such as RocketMQ, Kafka

Storage middleware. For example, cache, database, etc., such as Mysql, Redis. It is worth mentioning that, because storage-related systems are generally very complex, especially in the field of distributed storage, the system is more complicated, in Ali generally regard the database and cache storage type products as the existence of middleware level, so if someone says Mysql and Redis are not middleware, there is nothing to argue about.

Storage Proxy. Typical examples are ShardingSphere.

Gateway. Examples include Spring Cloud Gateway, Kong, Nginx.

ServiceMesh。Envoy and Istio are also divided into middleware departments on Ali's side.

In fact, it can be found that middleware actually does not have a clear definition of which open source products can be middleware and which are not.

After listing these typical middleware, we continue to discuss the theme of this section. What qualities do a middleware developer need?

Language basis. From a Java programmer's perspective, the basics are usually: collections, concurrency, JVM, common tool classes.

Operating system basics. Middleware developers often deal with operating systems, so computer basics are essential. I'll list some keywords for your reference

File IO. For example, pageCache, mmap, direct IO and other concepts.

Process threads. For example, green thread, coordination and other concepts.

Memory/CPU. For example, cgroup, cache line, bound core, etc.

Network Foundation. It can be found that each of the above middleware is inseparable from network communication. You must be familiar with the principles of TCP and HTTP. At the framework level, you need to be familiar with common network frameworks/tools such as NIO, Netty, GRPC, HttpClient, etc.

Distributed knowledge. Understand CAP, paxos, raft, zab, 2pc/3pc, base and other theoretical knowledge, for example, I saw some freshmen resume in a project experience is very interesting: according to MIT course Lab to implement Raft protocol POC.

Source code reading ability. I think the ability to read source code is a middleware developer essential quality, often can see a variety of source code analysis articles on the Internet, by reading the source code of open source middleware, you can learn from other people's design concepts, improve their own coding standards.

Keep the technology warm and embrace change. Middleware technology is changing with each passing day, and it is possible that a middleware is not defeated by similar products, but by the overall environment. For example, in recent years, cloud native fires, almost all middleware is embracing changes and actively aligning to K8s. Under this background, middleware developers need to have basic cognitive capabilities of K8s and be familiar with the basic concepts of pod, service, deployment, statefulSet and operator.

How to Become a Middleware Developer

After reading these requirements, some students may start to be speechless, but in fact it is not so terrible, this is the same as the earliest learning Java foundation, a lot of things have not been touched at first, think it is difficult, but familiar with it will find that it is also the case.

In my technical exchange group, there are often students complaining that they can only contact CRUD at ordinary times, but they can't contact these "tall" technologies at all. What I'm trying to say is, you're looking for opportunities. I have a few practical suggestions here.

Participate in open source community projects and contribute code. The best way to learn about middleware is to contribute it. Studying the source code with questions is the recommended way. All you need to do is find a suitable issue and solve it. Repeat this process and you are a middleware developer!

Do more hands-on experiments. Many of the qualities of middleware development applications mentioned above can be learned through hands-on experiments, such as hands-on implementation of a simple RPC framework, implementation of a Raft protocol POC, and comparison of FileChannel and MMAP performance through benchmarks. Believe me, this is much more useful than reading books, watching videos, and reading blogs.

Participate in the Middleware Challenge. At the beginning, Ali held an annual middleware performance challenge, and later other companies such as Huawei began to follow suit. Participating in these challenges can also accumulate a lot of experience, and you can also make a lot of friends by forming teams.

"What is middleware development" content is introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.