Shulou(Shulou.com)05/31 Report--

Envoy source code is how to analyze Dispatcher, I believe that many inexperienced people do not know what to do, so this article summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

Dispatcher

Dispatcher can be seen everywhere in the code of Envoy, and it can be said that it plays an important role in Envoy. A Dispatcher is an EventLoop, which undertakes the core functions such as task queue, network event processing, timer, signal processing and so on. The EventLoop (Each worker thread runs a "non-blocking" event loop) mentioned in the Envoy threading model article refers to this Dispatcher object. This part of the code is relatively independent and less coupled with other modules, but the importance is self-evident. The following is a class diagram related to Dispatcher, and the key concepts are introduced next.

Cdn.nlark.com/lark/0/2018/png/2826/1538970258265-1c4ce7cc-9283-48c4-81d2-ed4c595afb69.png ">

Dispatcher and Libevent

Dispatcher is essentially an EventLoop,Envoy that is not reimplemented, but reuses event_base in Libevent, encapsulates twice on the basis of Libevent and abstracts some event classes, such as FileEvent, SignalEvent, Timer and so on. Libevent is a C library, while Envoy is a C structure. In order to avoid manually managing the memory of these C structures, Envoy re-encapsulates the C structures exposed by libevent by inheriting unique_ptr.

Template class CSmartPtr: public std::unique_ptr {public: CSmartPtr (): std::unique_ptr (nullptr, deleter) {} CSmartPtr (T* object): std::unique_ptr (object, deleter) {}}

Through CSmartPtr, you can automatically manage the memory of some C data structures in Libevent through the RAII mechanism, as follows:

Extern "C" >

In Libevent, no matter the timer expires, the signal received, or the file can be read or written, it is an event, which is expressed uniformly by event type, while in Envoy, event is regarded as a member of ImplBase, and then all the objects of event type inherit ImplBase, thus realizing the abstraction of events.

Class ImplBase {protected: ~ ImplBase (); event raw_event_;}; SignalEvent

The implementation of SignalEvent is simple, initializing the event through evsignal_assign, and then adding the event through evsignal_add to make the event pending (see appendix for Libevent event status).

Class SignalEventImpl: public SignalEvent, ImplBase {public: / / signal_num: the signal value to be set / / cb: signal event handler SignalEventImpl (DispatcherImpl& dispatcher, int signal_num, SignalCb cb); private: SignalCb cb_;} SignalEventImpl::SignalEventImpl (DispatcherImpl& dispatcher, int signal_num, SignalCb cb): cb_ (cb) {evsignal_assign (& raw_event_, & dispatcher.base (), signal_num, [] (evutil_socket_t, short, void* arg)-> void {static_cast (arg)-> cb_ ();}, this); evsignal_add (& raw_event_, nullptr) } Timer

The Timer event exposes two interfaces, one for shutting down Timer and the other for starting Timer, which requires passing a time to set the expiration interval for Timer.

Class Timer {public: virtual ~ Timer () {} virtual void disableTimer () PURE; virtual void enableTimer (const std::chrono::milliseconds& d) PURE;}

When creating a Timer, the event is initialized through evtimer_assgin. At this time, the event is still pending and will not be triggered. It needs to be added to the Dispatcher through event_add before it can be triggered.

Class TimerImpl: public Timer, ImplBase {public: TimerImpl (Libevent::BasePtr& libevent, TimerCb cb); / / Timer void disableTimer () override; void enableTimer (const std::chrono::milliseconds& d) override;private: TimerCb cb_;}; TimerImpl::TimerImpl (DispatcherImpl& dispatcher, TimerCb cb): cb_ (cb) {ASSERT (cb_) Evtimer_assign (& raw_event_, & dispatcher.base (), [] (evutil_socket_t, short, void* arg)-> void {static_cast (arg)-> cb_ ();}, this);}

When disableTimer is called, it will call event_del to delete the event and make the event non-pending, while when enableTimer is called, it will indirectly call event_add to make the event a pending state, so that the timeout event will be triggered once the timeout time has expired.

Void TimerImpl::disableTimer () {event_del (& raw_event_);} void TimerImpl::enableTimer (const std::chrono::milliseconds& d) {if (d.count () = = 0) {event_active (& raw_event_, EV_TIMEOUT, 0);} else {std::chrono::microseconds us = std::chrono::duration_cast (d); timeval tv; tv.tv_sec = us.count () / 1000000 Tv.tv_usec = us.count ()% 1000000; event_add (& raw_event_, & tv);}}

The above code is not elegant in calculating the timer time timeval. Unreadable numeric constants such as 1000000 should be avoided. Some people in the community have suggested that they should be changed to the following form.

Auto secs = std::chrono::duration_cast (d); auto usecs = std::chrono::duration_cast (d-secs); tv.tv_secs = secs.count (); tv.tv_usecs = usecs.count (); FileEvent

Events related to socket sockets are encapsulated as FileEvent, which exposes two APIs: activate is used to actively trigger events. Typical usage scenarios such as wake-up EventLoop and Write Buffer with data can actively trigger writable events (typical usage scenarios in Envoy). SetEnabled is used to set the event type and add events to EventLoop to make it a pending state.

Void FileEventImpl::activate (uint32_t events) {int libevent_events = 0; if (events & FileReadyType::Read) {libevent_events | = EV_READ;} if (events & FileReadyType::Write) {libevent_events | = EV_WRITE;} if (events & FileReadyType::Closed) {libevent_events | = EV_CLOSED;} ASSERT (libevent_events); event_active (& raw_event_, libevent_events, 0) } void FileEventImpl::setEnabled (uint32_t events) {event_del (& raw_event_); assignEvents (events); event_add (& raw_event_, nullptr);} task queue

There is a task queue inside the Dispatcher, and a thread is created to handle the tasks in the task queue. Through the post method of Dispatcher, tasks can be delivered to the task queue and processed by threads in Dispatcher.

Void DispatcherImpl::post (std::function callback) {bool do_post; {Thread::LockGuard lock (post_lock_); do_post = post_callbacks_.empty (); post_callbacks_.push_back (callback);} if (do_post) {post_timer_- > enableTimer (std::chrono::milliseconds (0));}}

The post method adds the task represented by the passed callback to the member table variable of type vector represented by post_callbacks_. If post_callbacks_ is empty, the processing thread behind it is inactive, and it is awakened by setting a timeout time of 0 by post_timer_. When post_timer_ is constructed, the corresponding callback is set to runPostCallbacks, and the corresponding code is as follows:

DispatcherImpl::DispatcherImpl (TimeSystem& time_system, Buffer::WatermarkFactoryPtr&& factory):. Post_timer_ (createTimer ([this] ()-> void {runPostCallbacks ();})), current_to_delete_ (& to_delete_1_) {RELEASE_ASSERT (Libevent::Global::initialized (), ");}

RunPostCallbacks is a while loop that takes a task represented by callback from the post_callbacks_ each time to run it until post_callbacks_ is empty. Each time you run runPostCallbacks, you make sure that all the tasks are done. Obviously, if a new task is entered by the post during the execution of the runPostCallbacks by the thread, the new task can be appended directly to the end of the post_callbacks_ without the need to wake up the thread.

Void DispatcherImpl::runPostCallbacks () {while (true) {std::function callback; {Thread::LockGuard lock (post_lock_); if (post_callbacks_.empty ()) {return;} callback = post_callbacks_.front (); post_callbacks_.pop_front ();} callback ();}} DeferredDeletable

Finally, let's talk about DeferredDeletable, which is difficult to understand and important in Dispatcher, which is an empty interface from which all objects to be destructed inherit. Classes that inherit from DeferredDeletable like the following can be found everywhere in Envoy's code.

Class DeferredDeletable {public: virtual ~ DeferredDeletable () {}}

So what is delayed destructing? Which scene should it be used in? Delayed destructing refers to handing over the destructing action to Dispatcher, so DeferredDeletable and Dispatcher are closely related. The Dispatcher object has a vector that holds all the objects you want to defer destructing.

Class DispatcherImpl: public Dispatcher {. Private:. Std::vector to_delete_1_; std::vector to_delete_2_; std::vector* current_to_delete_;}

To_delete_1_ and to_delete_2_ are used to store all objects that need to be destructed. Here two vector are used to store them. Why do you want to do this? Current_to_delete_ always points to the list of objects currently being destructed, and each time it is destructed, it alternately points to another list of objects, alternating back and forth.

Void DispatcherImpl::clearDeferredDeleteList () {ASSERT (isThreadSafe ()); std::vector* to_delete = current_to_delete_; size_t num_to_delete = to_delete- > size (); if (deferred_deleting_ | |! num_to_delete) {return;} ENVOY_LOG (trace, "clearing deferred deletion list (size= {})", num_to_delete) If (current_to_delete_ = = & to_delete_1_) {current_to_delete_ = & to_delete_2_;} else {current_to_delete_ = & to_delete_1_;} deferred_deleting_ = true; for (size_t I = 0; I

Clear (); deferred_deleting_ = false;}

The above code uses to_delete to point to the list of objects currently being destructed when performing object destructing, and then points the current_to_delete_ to another list, so that objects can be safely added to the list when adding objects that are delayed to be deleted. Because deferredDelete and clearDeferredDeleteList are both running in the same thread, current_to_delete_ is a common pointer that can safely change the pointer to another without worrying about thread safety issues.

Void DispatcherImpl::deferredDelete (DeferredDeletablePtr&& to_delete) {ASSERT (isThreadSafe ()); current_to_delete_- > emplace_back (std::move (to_delete)); ENVOY_LOG (trace, "item added to deferred deletion list (size= {})", current_to_delete_- > size ()); if (1 = = current_to_delete_- > size ()) {deferred_delete_timer_- > enableTimer (std::chrono::milliseconds (0));}}

When there is an object to be destructed, just call deferredDelete. Inside this function, you will put the object into the list to be destructed through current_to_delete_, and finally determine whether the size of the list to be destructed is 1. If 1 indicates that this is the first time to add an object for deferred destructing, then you need to wake up the thread behind to execute the clearDeferredDeleteList function through deferred_delete_timer_. The reason for doing this is to avoid multiple wakes, because there is a situation where the thread has awakened to execute the clearDeferredDeleteList, and in the process there are other objects that need to be destructed to add to the vector.

So far, the implementation principle of deferredDelete is basically analyzed, and we can see that its implementation is very similar to the implementation of the task queue, except that one is to cycle through the tasks represented by callback, and the other is to destruct the object. Finally, let's take a look at the application scenario of deferredDelete, but "Why delay destructing?" Code snippets like the following are often seen in the source code of Envoy.

ConnectionImpl::ConnectionImpl (Event::Dispatcher& dispatcher, ConnectionSocketPtr&& socket, TransportSocketPtr&& transport_socket, bool connected) {. } / / pass bare pointer to callback file_event_ = dispatcher_.createFileEvent (fd (), [this] (uint32_t events)-> void {onFileEvent (events);}, Event::FileTriggerType::Edge, Event::FileReadyType::Read | Event::FileReadyType::Write);.}

The callback passed to Dispatcher is called back through a bare pointer. If the object is already destructed during the callback, the problem of wild pointer will occur. I believe students with a reasonable level of C++ will see this problem, unless they can logically guarantee that the life cycle of Dispatcher is shorter than all objects, which ensures that the object will not be destructed during the callback, but this is impossible. Because Dispatcher is the core of EventLoop.

A thread runs an EventLoop until the end of the thread before the Dispatcher object is destructed, which means that the life cycle of the Dispatcher object is the longest. So there is no logical guarantee that the object will not be destructed during the callback. Some people may wonder, wouldn't it be possible to avoid the problem of wild pointers by canceling the registered event when the object is destructed? What if the event has been triggered and callback is waiting to run? Or is callback halfway running? The former libevent can be guaranteed, when calling event_del, you can cancel the events waiting to run, but the latter is powerless, at this time, if the object is destructed, the behavior is undefined. Thinking along this line of thinking, is it possible to solve the problem by ensuring that no callback is running when the object is destructed? Yes, just make sure that all the callback in execution is finished, and then do object destructing. You can take advantage of the fact that Dispatcher executes all callback sequentially, and inserting a task into Dispatcher is used to destruct objects, so when this task is executed, you can ensure that no other callback is running. Through this method, the problem of wild pointer encountered here is solved perfectly.

Some people may wonder, can we use shared_ptr and shared_from_this to solve this here? Yes, this is a secret weapon to solve the problem of object destructing in a multi-threaded environment. By prolonging the life cycle of the object, extending the life cycle of the object to the same as callback, and then destructing after callback execution, we can also achieve the effect, but this brings two problems. The first is that the life cycle of the object is infinitely lengthened. Although delayed destructing also lengthens the life cycle, the time is predictable. Once the EventLoop executes the clearDeferredDeleteList task, it will be recycled immediately. The life cycle of the shared_ptr depends on when the callback runs, and there is no guarantee when the callback will run. For example, a readable event waiting for socket is called back. If the peer does not send data, then the callback will not be run and the object cannot be destructed all the time. Accumulation over a long period of time will lead to an increase in memory utilization. The second is that it is more intrusive in the way it is used, and it is necessary to force the use of shared_ptr to create objects.

Generally speaking, the implementation of Dispatcher is relatively simple and clear, and it is easy to verify its correctness, and its function is also relatively weak. It is similar to chromium's MessageLoop and boost's asio, but its function is much worse. Fortunately, this is specially designed for Envoy, and the scene of Envoy is relatively simple, so it doesn't have to be so general-purpose. Another thing that I find strange is why the implementation of DeferredDeletable should be stored alternately with two queues of to_delete_1_ and to_delete_2_. In fact, according to my understanding, a queue can be used, because clearDeferredDeleteList and deferredDelete are guaranteed to be executed in the same thread, just like Dispatcher's task queue, all tasks to be executed are stored in a queue and executed in a loop. But this is not done in Envoy. I understand that the reason for this design may be that the importance of delayed destructing is lower than that of task queues. If the destructions of a large number of objects are kept in a queue and destructed cyclically, it will inevitably affect the execution of other critical tasks, so here they are split into two queues, and multiple tasks are executed alternately. It's like splitting a large task into several small tasks to execute.

After reading the above, have you mastered how the Envoy source code analyzes Dispatcher? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.