Shulou(Shulou.com)06/03 Report--

This article focuses on "how to understand that Go Map and Slice belong to non-linear security". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to understand that Go Map and Slice are non-linear security.

An example of nonlinear security

Slice

We use multiple goroutine to manipulate variables of type slice to see what happens.

As follows:

Func main () {var s [] string for i: = 0; I < 9999; iFried + {go func () {s = append (s, "fried fish in brain")} ()} fmt.Printf ("fried fish only in% d", len (s))}

Output result:

/ / 5790 fried fish in the first execution / / 7370 fried fish in the second execution / / 6792 fried fish in the third execution

You will find that no matter how many times you execute, the probability of each output will not be the same. That is, the value appended to slice, overwriting occurs.

So the amount appended in the loop is not equal to the final value. And this kind of situation, will not report wrong, is an implicit problem whose occurrence rate is not high.

The main reason for this is that there is something wrong with the program logic itself, and reading the same index bits will naturally lead to overwritten writes.

Map

The same is true for map. Repeat writes for variables of type map.

As follows:

Func main () {s: = make (map [string] string) for I: = 0; I < 99; iFry + {go func () {s ["fried fish"] = "sucking fish"} ()} fmt.Printf ("fried fish only", len (s))}

Output result:

Fatal error: concurrent map writes goroutine 18 [running]: runtime.throw (0x10cb861, 0x15) / usr/local/Cellar/go/1.16.2/libexec/src/runtime/panic.go:1117 + 0x72 fp=0xc00002e738 sp=0xc00002e708 pc=0x1032472 runtime.mapassign_faststr (0x10b3360, 0xc0000a2180, 0x10c91da, 0x6 0x0) / usr/local/Cellar/go/1.16.2/libexec/src/runtime/map_faststr.go:211 + 0x3f1 fp=0xc00002e7a0 sp=0xc00002e738 pc=0x1011a71 main.main.func1 (0xc0000a2180) / Users/eddycjy/go-application/awesomeProject/main.go:9 + 0x4c fp=0xc00002e7d8 sp=0xc00002e7a0 pc=0x10a474c runtime.goexit () / usr/local/Cellar/go/1.16.2/libexec/src/runtime/asm_amd64.s:1371 + 0x1 fp=0xc00002e7e0 sp=0xc00002e7d8 pc=0x1063fe1 created by main.main / Users/eddycjy/go-application/awesomeProject/main.go:8 + 0x55

Boy, the program will report an error directly. And it is the fatal error caused by the Go source code calling the throw method, that is, the Go process will be interrupted.

I have to say that the fatal error: concurrent map writes error message caused by this concurrent write map. I have a friend who has seen it dozens of times, different groups, different people.

It's an implicit question from the Nikkei.

How to support concurrent read and write

Lock the map

In fact, we still have the demand to read and write map concurrently (program logic decision), because goroutine in the Go language is so convenient.

For example, when you write a crawler task, you will basically use multiple goroutine, get the data and then write it to map or slice.

Go officials provide a simple and convenient way to do this in Go maps in action:

Var counter = struct {sync.RWMutex map [string] int} {m: make (map [string] int)}

This statement declares a variable, which is an anonymous structure (struct) body that contains a native and an embedded read-write lock sync.RWMutex.

To read data from a variable, call the read lock:

Counter.RLock () n: = counter.m ["fried fish"] counter.RUnlock () fmt.Println ("fried fish:", n)

To write data to a variable, call the write lock:

Counter.Lock () counter.m ["fried fish"] + + counter.Unlock ()

This is one of the most common ways in which Map supports reading and writing.

Sync.Map

Preface

Although there is a minimalist scheme of Map+Mutex, there are still some problems. That is, when the amount of data in map is very large, only one Mutex is very scary, and a lock will lead to a large number of competing locks, leading to various conflicts and poor performance.

A common solution is fragmentation, which divides a large map into multiple intervals, and each interval uses multiple locks, so that the granularity of sublocks is greatly reduced. However, the implementation of the scheme is very complex and error-prone. Therefore, the Go team did not recommend it until the comparison, but adopted other options.

This scheme is the sync.Map supported in Go1.9, which supports concurrent reading and writing of map, which plays a supplementary role.

Specific introduction

Sync.Map of Go language supports concurrent read and write map, adopts the mechanism of "space for time", and redundant two data structures, namely: read and dirty, to reduce the impact of locking on performance:

Type Map struct {mu Mutex read atomic.Value / / readOnly dirty map [interface {}] * entry misses int}

It is specially designed for append-only scenarios, that is, scenarios that read more and write less. This is one of his strengths.

If there are many writes / concurrency scenarios, it will lead to the invalidation of read map cache, the need to lock, the increase of conflicts and the sharp decline in performance. This is his major weakness.

The following common methods are provided:

Func (m * Map) Delete (key interface {}) func (m * Map) Load (key interface {}) (value interface {}, ok bool) func (m * Map) LoadAndDelete (key interface {}) (value interface {}, loaded bool) func (m * Map) LoadOrStore (key, value interface {}) (actual interface {}, loaded bool) func (m * Map) Range (f func (key, value interface {}) bool) func (m * Map) Store (key, value interface {})

Delete: delete the value of a key.

Load: returns the value of the key stored in map, or nil if there is no value. The ok result indicates whether the value was found in the map.

LoadAndDelete: deletes the value of a key and returns the previous value, if any.

LoadOrStore: returns the existing value of the key, if it exists. Otherwise, it stores and returns the given value. The result of the load is true if the value is loaded, or false if it is stored.

Range: a recursive call that calls the closure function f for each key and value that exists in map. Stop iterating if f returns false.

Store: stores and sets the value of a key.

The actual operation examples are as follows:

Var m sync.Map func main () {/ / write data: = [] string {"fried fish", "salted fish", "grilled fish", "steamed fish"} for I: = 0; I < 4 Read + {go func (I int) {m.Store (I, data [I])} (I)} time.Sleep (time.Second) / / read v, ok: = m.Load (0) fmt.Printf ("Load:% v,% v\ n", v, ok) / delete m.Delete (1) / / read or write v, ok = m.LoadOrStore (1, "suck fish") fmt.Printf ("LoadOrStore:% v") % v\ n ", v, ok) / / traversing m.Range (func (key, value interface {}) bool {fmt.Printf (" Range:% v,% v\ n ", key, value) return true})}

Output result:

Load: fried fish, true LoadOrStore: sucked fish, false Range: 0, fried fish Range: 1, sucked fish Range: 3, steamed fish Range: 2, grilled fish why not support

Go Slice, mainly index bit overwrite problem, this does not need to be entangled, is bound to program logic in the preparation of obvious defects, self-improvement is good.

But Go map is different, many people think that it is supported by default, one accidentally overturned, so common. Then why the Go official does not support, is it too complicated, the performance is too poor, in the end why?

The reasons are as follows (via @ go faq):

Typical usage scenario: a typical map usage scenario is that there is no need for secure access from multiple goroutine.

Atypical scenarios (requiring atomic operations): map may be part of larger data structures or calculations that have been synchronized.

Performance scenario considerations: if you only add security to a small number of programs, resulting in all map operations have to deal with mutex, it will degrade the performance of most programs.

To sum up, after a long discussion, Go officials decided that Go map should be more suitable for typical usage scenarios, rather than for a small number of cases, resulting in the cost (performance) of most programs and decided not to support it.

At this point, I believe you have a deeper understanding of "how to understand that Go Map and Slice are non-linear security". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.