Shulou(Shulou.com)06/03 Report--

This article mainly explains "how to deeply understand thread pool". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "how to deeply understand the thread pool".

This article will introduce the principle of thread pool from the following aspects.

Why use thread pools?

How thread pools work

Two ways for thread pool to submit tasks

Analysis of ThreadPoolExecutor Source Code

Answer the question at the beginning

Best practices for thread pools

Summary

I believe that after reading the understanding of the thread pool will be further, liver text is not easy, do not finish three times after reading.

Why use thread pools?

As mentioned above, there are three major overhead for creating a thread, as follows:

1. In fact, the thread model in Java is based on the operating system native thread model, that is to say, threads in Java are actually implemented based on kernel threads. Thread creation, destructing and synchronization all require system calls, while system calls need to switch back and forth between the user mode and the kernel, which is relatively expensive. The life cycle of a thread includes "thread creation time", "thread execution time", "thread destruction time", creation and destruction all need to cause system calls. 2. Each Thread needs to be supported by a kernel thread, which means that each Thread needs to consume certain kernel resources (such as the stack space of the kernel thread). Therefore, the number of Thread that can be created is limited. By default, the thread stack size of a thread is 1m.

As shown in the figure, under Java 8, creating 19 threads (thread # 19) requires the creation of 19535 KB, or about 1m. Reserved represents that if 19 threads are created, the operating system guarantees that so much space will be allocated (not necessarily), and committed represents the actual amount of space allocated.

Voiceover: note that this is a thread footprint under Java 8, but in Java 11, threads are greatly optimized. It takes only about 40 KB to create a thread, which greatly reduces space consumption.

3. There are too many threads, resulting in context switching overhead that can not be ignored.

Thus it can be seen that the creation of threads is expensive, so they must be managed in the form of a thread pool, and the thread size and management threads must be reasonably set in the thread pool in order to create a reasonable thread size to maximize benefits and minimize risks. for developers, they don't have to care about how threads are created, how to destroy, and how to cooperate. You only need to care about when the submitted task will be completed, and all the trivial tasks such as thread tuning and monitoring will be handed over to the thread pool, so it will be a great relief for developers!

The idea of pooling similar to thread pool is applied in many places, such as database connection pool, Http connection pool, etc., which avoids the creation of expensive resources, improves performance, and liberates developers.

ThreadPoolExecutor Design Architecture Diagram

First, let's take a look at the design diagram of the Executor framework.

Executor: the top-level Executor interface only provides an execute interface, which realizes the decoupling between submitting tasks and executing tasks. This method is the core and the focus of our source code analysis. This method is finally implemented by ThreadPoolExecutor.

ExecutorService extends the Executor interface to implement methods such as terminating executors, single / batch submission of tasks, etc.

AbstractExecutorService implements the ExecutorService interface, implements all methods except execute, and gives only one of the most important execute methods to ThreadPoolExecutor implementation.

Although such a hierarchical design looks like a lot of levels, but each layer of each department, the logic is clear, it is worth learning.

How thread pools work

First, let's look at how to create a thread pool.

ThreadPoolExecutor threadPool = new ThreadPoolExecutor (10,20,600L, TimeUnit.SECONDS, new LinkedBlockingQueue (4096), new NamedThreadFactory ("common-work-thread")); / / set reject policy, default is AbortPolicy threadPool.setRejectedExecutionHandler (new ThreadPoolExecutor.AbortPolicy ())

Take a look at the construction method signature as follows

Public ThreadPoolExecutor (int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit, BlockingQueue workQueue, ThreadFactory threadFactory RejectedExecutionHandler handler) {/ / omit several codes}

To understand the meaning of these parameters, you must have a clear understanding of the process of submitting and executing tasks in the thread pool, as follows

The picture is from Meituan's technical team

The steps are as follows

1. CorePoolSize: if the thread is still running after the task is submitted, when the number of threads is less than the corePoolSize value, threads in the thread pool will be created regardless of whether the thread in the pool is busy or not, and the task will be handed over to the newly created thread for processing. If the number of threads is less than or equal to corePoolSize, then these threads will not be recycled unless allowCoreThreadTimeOut is set to true, but generally will not do so, because frequent creation and destruction of threads will greatly increase the overhead of system calls.

2. WorkQueue: if the number of threads is greater than the number of cores (corePoolSize) and less than the maximum number of threads (maximumPoolSize), the task will be thrown into the blocking queue first, and then the thread itself will block the execution of the task in the queue.

3. MaximumPoolSize: the maximum number of threads that can be created in the thread pool. If the queue is full when the task is submitted and the number of threads does not reach this set value, the thread will be created and the submitted task will be executed. If the queue is full but the number of thread pools has reached this value when the task is submitted, it means that the load capacity of the thread pool has been exceeded, and the rejection policy will be implemented. We can't let a steady stream of tasks come in and crush the thread pool. First of all, we have to make sure that the thread pool works properly.

4. RejectedExecutionHandler: there are four rejection strategies

AbortPolicy: discard the task and throw an exception, which is also the default policy

CallerRunsPolicy: the caller's thread is used to execute the task, so the opening question "must be returned immediately after the thread leaves the task in the thread pool?" We can answer that if the CallerRunsPolicy policy is used, the thread that submits the task (such as the main thread) cannot guarantee that it will return immediately after submitting the task, and when the reject policy is triggered, it has to deal with the task in person.

DiscardOldestPolicy: discards the first task in the blocking queue and executes the current task.

DiscardPolicy: simply discard tasks without throwing any exceptions. This strategy only applies to unimportant tasks.

5. KeepAliveTime: thread survival time. If threads exceeding the corePoolSize size are in idle state during this time, these threads will be reclaimed.

6. ThreadFactory: you can use this parameter to set the name of the thread pool, specify defaultUncaughtExceptionHandler (what's the use, described later), and even set the thread as a daemon thread.

Now the question is, how to set these parameters reasonably.

First, let's take a look at the thread size setting.

Tell us that there should be two situations.

1. For CPU-intensive tasks, on systems with Ncpu processors, optimal utilization is usually achieved when the thread pool size is Ncpu + 1. + 1 is because when computing-intensive threads occasionally pause work due to page faults or other reasons, this "extra" thread also ensures that CPU clock cycles are not wasted. The so-called CPU intensive means that threads have been busy all the time. In this way, setting the size of the thread pool to Ncpu + 1 avoids thread context switching, keeps the thread busy all the time, and maximizes CPU utilization.

two。 For IO-intensive tasks, it also gives the following formula

Just take a look at these formulas, which are basically not used in actual business scenarios. These formulas are too theoretical, divorced from business scenarios, and can only be used as a theoretical reference. For example, you say that CPU-intensive tasks set the thread pool size to N + 1, but in fact, there is often more than one thread pool in the business, and the formula applied in this situation is confused.

Let's look at the size setting of workQueue.

As can be seen from the above, if the maximum thread is greater than the number of core threads, new threads will be added only if and only if the core thread is full and workQueue is also full, that is to say, if workQueue is an unbounded queue, then when the number of threads increases to corePoolSize, there will never be any new threads, that is, the setting of maximumPoolSize will be invalid, and the RejectedExecutionHandler reject policy will not be triggered, and tasks will only be continuously populated to workQueue until OOM.

So workQueue should be bounded queues, at least to ensure that the thread pool can work properly if the task is overloaded, so which are bounded queues and which are unbounded queues.

The following two commonly used bounded queues

LinkedBlockingQueue: a bounded queue consisting of linked lists that arranges elements in first-in, first-out (FIFO) order, but note that you need to specify their size when creating them, otherwise their size defaults to Integer.MAX_VALUE, which is equivalent to an unbounded queue.

ArrayBlockingQueue: a bounded queue implemented by an array that arranges elements in first-in, first-out (FIFO) order.

Unbounded queue We often use PriorityBlockingQueue as a priority queue. When tasks are inserted, we can specify their weights to allow them to be executed first, but this queue is rarely used for the simple reason that the execution order of tasks in the thread pool is generally equal. If there is a need for certain types of tasks to be executed first, open a thread pool and isolate different task types with different thread pools. It is also a practice of making rational use of thread pools.

Speaking of which, I'm sure you can answer the opening question, "Why doesn't the Ali Java code specification allow you to use Executors to quickly create thread pools? The most common are the following two ways to create

Image-20201109002227476

The maximum number of threads for the newCachedThreadPool method is set to Integer.MAX_VALUE, while the LinkedBlockingQueue does not declare the size when the newSingleThreadExecutor method creates the workQueue, which is equivalent to creating an unbounded queue, which can lead to OOM if you are not careful.

How to set up threadFactory

In general, there will be multiple thread pools in the business. If there is a problem with a thread pool, it is important to locate which thread has the problem, so it is necessary to give each thread pool a name. Our company uses the NamedThreadFactory of dubbo to generate threadFactory, and it is very easy to create.

New NamedThreadFactory ("demo-work")

Its implementation is still very ingenious, you can take a look at its source code, each call, there is a counter at the bottom will be increased by one, will be named "demo-work-thread-1", "demo-work-thread-2", "demo-work-thread-3" such an incremental string.

In actual business scenarios, it is generally difficult to determine the size of corePoolSize and workQueue,maximumPoolSize. If something goes wrong, generally speaking, you can only reset these parameters before release, which often takes some time. Meituan's article gives an eye-catching solution. When problems are found (thread pool monitoring alarm), adjust these parameters dynamically to make them take effect in real time. It is indeed a good idea to solve problems in time when they are found.

Two ways for thread pool to submit tasks

Once the thread pool has been created, there are two ways to submit tasks to it. Call the execute and submit methods to see the method signatures of these two methods.

/ / method 1: execute method public void execute (Runnable command) {} / / Mode 2: three methods of submit in ExecutorService: Future submit (Callable task); Future submit (Runnable task, T result); Future submit (Runnable task)

The difference is that calling execute has no return value, while calling submit can return Future, so what can this Future do? look at its interface.

Public interface Future {/ * cancel the task in progress, return false if the task has been executed or cancelled, or cannot be cancelled for some reason * if the task has not started or the task has started but can be interrupted (mayInterruptIfRunning is true), then * you can cancel / interrupt this task * / boolean cancel (boolean mayInterruptIfRunning) / * whether the task has been cancelled before completion * / boolean isCancelled (); / * normal execution of the process, or throw an exception, or cancel the completion of the task will return true * / boolean isDone () / * blocking the execution result of waiting task * / V get () throws InterruptedException, ExecutionException / * blocks waiting for the execution result of the task, but the time is specified here. If the task has not been completed within the timeout time, * an TimeoutException exception is thrown * / V get (long timeout, TimeUnit unit) throws InterruptedException, ExecutionException, TimeoutException;}

You can use Future to cancel a task, determine whether it has been cancelled / completed, or even block waiting for a result.

Why can submit submit Runnable and return the execution result of Future at the same time?

It turns out that before the final execution of execute, task is encapsulated into RunnableFuture,newTaskFor with newTaskFor and the class FutureTask is returned. The structure diagram is as follows.

You can see that FutureTask implements both the Runnable interface and the Future interface, so when submitting a task, you can also use the Future interface to cancel the task, get the status of the task, and wait for the results of these operations.

In addition to whether execute and submit can return the execution result, there is another important difference, that is, using execute to execute if an exception occurs, it cannot be caught, and the uncaughtException method of ThreadGroup will be executed by default (the logic corresponding to the number 2 below).

So if you want to monitor the exceptions that occur during the execution of the execute method, you need to specify a UncaughtExceptionHandler through threadFactory, so that the 1 in the figure above will be executed, and then the logic in UncaughtExceptionHandler will be executed, as shown below:

/ / 1. Implement your own thread pool factory ThreadFactory factory = (Runnable r)-> {/ / create a thread Thread t = new Thread (r) / / set the default logic t.setDefaultUncaughtExceptionHandler ((Thread thread1, Throwable e)-> {/ / set the statistical monitoring logic System.out.println ("thread factory set exceptionHandler + e.getMessage ());}) to the created thread to implement the exception in the thread object; return t;}; / / 2. Create a self-defined thread pool, using the self-defined thread factory ExecutorService service = new ThreadPoolExecutor (1, 1, 0, TimeUnit.MILLISECONDS,new LinkedBlockingQueue (10), factory); / / 3. Submit task service.execute (()-> {int item1Universe 0;})

Executing the above logic will eventually output the "thread factory set exceptionHandler/ by zero". In this way, our monitoring logic can be executed through the set defaultUncaughtExceptionHandler.

If you use submit, how to catch exceptions? when we call future.get, we can catch

Callable testCallable = xxx; Future future = executor.submit (myCallable); try {future1.get (3);} catch (InterruptedException e) {e.printStackTrace ();} catch (ExecutionException e) {e.printStackTrace ();}

So why does future catch asynchronism only when get is executed, because the exception is saved after an exception is thrown when submit is executed, and only when get is thrown?

This article about the implementation process of execute and submit why God is very thorough, I will not pick up the wisdom of others, I suggest you taste it well, the harvest will be very great!

Analysis of ThreadPoolExecutor Source Code

After laying the groundwork for so much, we have finally reached the core source code analysis link.

For thread pool, we are most concerned about its "state" and "number of threads that can run". Generally speaking, we can choose to record it with two variables, but Doug Lea uses only one variable (ctl) to achieve its goal. We know that the more variables, the worse the maintainability of the code, and the easier it is to produce bug, so only one variable is used to achieve the effect of two variables. This makes the code much more maintainable, so how did he design it?

/ / ThreadPoolExecutor.java public class ThreadPoolExecutor extends AbstractExecutorService {private final AtomicInteger ctl = new AtomicInteger (ctlOf (RUNNING, 0)); private static final int COUNT_BITS = Integer.SIZE-3; private static final int CAPACITY = (1 = CAPACITY | | wc > = (core? CorePoolSize: maximumPoolSize) return false; / / otherwise CAS increases the number of threads. If you successfully jump out of the double loop if (compareAndIncrementWorkerCount (c)) break retry; c = ctl.get () / / Re-read ctl / / if the running state of the thread changes, skip to the outer loop to continue to execute if (runStateOf (c)! = rs) continue retry; / / because CAS failed to increase the number of threads, continue to execute the inner loop of retry}} boolean workerStarted = false; boolean workerAdded = false Worker w = null; try {/ / can be executed here, which means that the conditions for adding worker are met, so create a worker and prepare to add it to the thread pool to execute tasks w = new Worker (firstTask); final Thread t = w.thread If (t! = null) {/ / is locked because w will be added to workers below. Workers is HashSet, not thread-safe, so lock is needed to guarantee final ReentrantLock mainLock = this.mainLock; mainLock.lock () Try {/ / check the state of the thread pool again to prevent interruptions such as int rs = runStateOf (ctl.get ()) / / if the thread pool status is less than SHUTDOWN (that is, RUNNING), / / or if the status is SHUTDOWN but firstTask = = null (means not receiving tasks, but only creating threads to process tasks in workQueue), the condition for adding worker is met (rs

< SHUTDOWN || (rs == SHUTDOWN && firstTask == null)) { // 如果线程已启动，显然有问题（因为创建 worker 后，还没启动线程呢），抛出异常 if (t.isAlive()) throw new IllegalThreadStateException(); workers.add(w); int s = workers.size(); // 记录最大的线程池大小以作监控之用 if (s >

LargestPoolSize) largestPoolSize = s; workerAdded = true;}} finally {mainLock.unlock ();} / / indicates that the worker has been added to the workers successfully, and the thread if (workerAdded) {t.start () is started. WorkerStarted = true;} finally {/ / failed to add threads, execute the addWorkerFailed method, mainly to remove worker from workers, reduce the number of threads, and try to close operations such as thread pool if (! WorkerStarted) addWorkerFailed (w);} return workerStarted;}

From this code we can see the unpredictability of the multithreaded situation, and we find that when the conditions are met, the thread state is re-check to prevent the operation of thread pool state changes such as interruptions, which also gives us inspiration: various critical conditions in the multithreaded environment must be considered in place.

After the execution of the addWorker creates the worker successfully, the thread starts to execute (t.start ()). Since the Worker itself is passed to the thread when the Worker is created, the run method of Worker is called after the thread is started

Public void run () {runWorker (this);}

You can see that the runWorker method will eventually be called, so let's analyze the runWorker method

Final void runWorker (Worker w) {Thread wt = Thread.currentThread (); Runnable task = w.firstTask; w.firstTask = null; / / unlock will call the tryRelease method to set state to 0, which means to allow interrupts, and the conditions for allowing interrupts have been mentioned above in interruptIfStarted (), that is, state > = 0 w.unlock (); boolean completedAbruptly = true Try {/ / if a thread is created when a task is submitted and the task is thrown to this thread, the task / / is executed first, otherwise the task is obtained from the task queue to execute (that is, the getTask () method) while (task! = null | | (task = getTask ())! = null) {w.lock () / / if the thread pool state is > = STOP (that is, STOP,TIDYING,TERMINATED), the thread should break / / if the thread pool state

< STOP, 线程不应该中断,如果中断了（Thread.interrupted() 返回 true，并清除标志位）,再次判断线程池状态（防止在清除标志位时执行了 shutdownNow() 这样的方法）,如果此时线程池为 STOP，执行线程中断 if ((runStateAtLeast(ctl.get(), STOP) || (Thread.interrupted() && runStateAtLeast(ctl.get(), STOP))) && !wt.isInterrupted()) wt.interrupt(); try { // 执行任务前，子类可实现此钩子方法作为统计之用 beforeExecute(wt, task); Throwable thrown = null; try { task.run(); } catch (RuntimeException x) { thrown = x; throw x; } catch (Error x) { thrown = x; throw x; } catch (Throwable x) { thrown = x; throw new Error(x); } finally { // 执行任务后，子类可实现此钩子方法作为统计之用 afterExecute(task, thrown); } } finally { task = null; w.completedTasks++; w.unlock(); } } completedAbruptly = false; } finally { // 如果执行到这只有两种可能，一种是执行过程中异常中断了，一种是队列里没有任务了，从这里可以看出线程没有核心线程与非核心线程之分，哪个任务异常了或者正常退出了都会执行此方法，此方法会根据情况将线程数-1 processWorkerExit(w, completedAbruptly); } } 来看看 processWorkerExit 方法是咋样的 private void processWorkerExit(Worker w, boolean completedAbruptly) { // 如果异常退出，cas 执行线程池减 1 操作 if (completedAbruptly) decrementWorkerCount(); final ReentrantLock mainLock = this.mainLock; mainLock.lock(); try { completedTaskCount += w.completedTasks; // 加锁确保线程安全地移除 worker workers.remove(w); } finally { mainLock.unlock(); } // woker 既然异常退出，可能线程池状态变了（如执行 shutdown 等），尝试着关闭线程池 tryTerminate(); int c = ctl.get(); // 如果线程池处于 STOP 状态，则如果 woker 是异常退出的，重新新增一个 woker，如果是正常退出的，在 wokerQueue 为非空的条件下，确保至少有一个线程在运行以执行 wokerQueue 中的任务 if (runStateLessThan(c, STOP)) { if (!completedAbruptly) { int min = allowCoreThreadTimeOut ? 0 : corePoolSize; if (min == 0 && ! workQueue.isEmpty()) min = 1; if (workerCountOf(c) >

= min) return; / / replacement not needed} addWorker (null, false);}}

Next, we analyze woker's method of fetching tasks from workQueue, getTask.

Private Runnable getTask () {boolean timedOut = false; / / Did the last poll () time out? For (;;) {int c = ctl.get (); int rs = runStateOf (c) / / if the thread pool status is at least STOP or / / thread pool status = = SHUTDOWN and the task queue is empty / / then reduce the number of threads and return null, in which case the runWorker analyzed above executes processWorkerExit so that the woker that gets this Task exits the if (rs > = SHUTDOWN & & (rs > = STOP | | workQueue.isEmpty () {decrementWorkerCount () Return null;} int wc = workerCountOf (c); / / if allowCoreThreadTimeOut is true, any thread in idle state in keepAliveTime time will be reclaimed. If the number of threads is greater than corePoolSize, boolean timed = allowCoreThreadTimeOut will be reclaimed if it is in idle state in keepAliveTime time | | wc > corePoolSize / / several conditions under which worker should be recycled, which is relatively simple, so skip if ((wc > maximumPoolSize | | (timed & & timedOut)) & & (wc > 1 | | workQueue.isEmpty () {if (compareAndDecrementWorkerCount (c)) return null; continue } try {/ / blocks the acquisition of task. If no task is obtained within the keepAliveTime time, the timedOut has timed out. In this case, timedOut is true Runnable r = timed? WorkQueue.poll (keepAliveTime, TimeUnit.NANOSECONDS): workQueue.take (); if (r! = null) return r; timedOut = true;} catch (InterruptedException retry) {timedOut = false;}

After the above source code analysis, I believe we have a good understanding of the working principle of the thread pool. Let's briefly take a look at some other useful methods. At the beginning, we mentioned the monitoring problem of the thread pool. Let's see which indicators can be monitored.

Int getCorePoolSize (): gets the number of core threads.

Int getLargestPoolSize (): historical peak number of threads.

Int getMaximumPoolSize (): maximum number of threads (thread pool thread capacity).

Int getActiveCount (): current number of active threads

Int getPoolSize (): total number of threads in the current thread pool

BlockingQueuegetQueue () the task queue of the current thread pool, from which you can get the total backlog of tasks, getQueue.size ()

The idea of monitoring is also very simple. Start a timed thread ScheduledThreadPoolExecutor and collect these thread pool indicators regularly, which is generally realized by some open source tools such as Grafana + Prometheus + MicroMeter.

How to realize the preheating of the core thread pool

Using the prestartAllCoreThreads () method, this method creates corePoolSize threads at once, without having to wait until the task is submitted, which can be processed as soon as the task is submitted.

How to dynamically adjust thread pool parameters

SetCorePoolSize (int corePoolSize) resizes the core thread pool

SetMaximumPoolSize (int maximumPoolSize)

SetKeepAliveTime () sets the survival time of the thread

Answer the question at the beginning

Other questions are basically answered in the source code analysis. Here are some other questions.

1. What's the difference between the thread pool implementation of Tomcat and the thread pool implementation of JDK? is there a thread pool implementation similar to Tomcat in Dubbo? There is something called EagerThreadPool in Dubbo. You can take a look at its instructions.

As you can see from the comments, if the core threads are in the busy state, if new requests come in, EagerThreadPool will choose to create the thread first rather than put it in the task queue, so that it can respond to those requests more quickly.

The Tomcat implementation is similar, but with a slight difference. When Tomcat starts, minSpareThreads threads are created first. If these threads are busy after a period of time, they are created each time in minSpareThreads steps, essentially to respond to processing requests more quickly. The specific source code can be seen in its ThreadPool implementation, which will not be expanded here.

2. There has been such a problem in our gateway dubbo call thread pool: during stress testing, the interface can return normally, but the interface RT is very high. Assuming that the core thread size is 500,800 and the buffer queue is 5000, can you find some problems in this setting and tune these parameters? This parameter can obviously see the problem, first of all, the task queue is set too large, after the task reaches the core thread, if another request comes in, it will first enter the task queue, and the queue is full before creating a thread, and it takes a lot of overhead to create a thread, so we later set the core thread to the same as the largest thread, and call prestartAllCoreThreads () to preheat the core thread, so we don't have to wait for the request to create a thread.

Several best practices for thread pools

1. The tasks performed by the thread pool should be independent of each other, and if they depend on each other, it may lead to deadlock, such as the following code

ExecutorService pool = Executors .newSingleThreadExecutor (); pool.submit (()-> {try {String qq=pool.submit (()-> "QQ"). Get (); System.out.println (qq);} catch (Exception e) {})

2. Core tasks and non-core tasks should be separated by multiple thread pools.

Once upon a time, there was such a failure in our business: suddenly many users reported that they could not receive text messages, and they found that texting was in a thread pool, and other timing scripts also used this thread pool to execute tasks. this script may generate hundreds of tasks a minute, resulting in almost no chance of execution in the online pool of texting. Later, we used two thread pools to separate texting from executing scripts to solve the problem.

3. Add thread pool monitoring and set thread pool dynamically

As mentioned earlier, it is difficult to determine the parameters of the thread pool at one time. Since it is difficult to determine and to ensure that the problem is solved in time, we need to increase monitoring for the thread pool to monitor the queue size, the number of threads, and so on. We can set an alarm within 3 minutes, for example, if the queue task is full all the time, so as to give an early warning. If online operations such as degradation are triggered due to unreasonable thread pool parameters, the number of core threads and the maximum number of threads can be modified in real time by dynamically setting the thread pool, and the problem can be repaired in time.

At this point, I believe you have a deeper understanding of "how to deeply understand the thread pool". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.