Shulou(Shulou.com)06/03 Report--

This article mainly explains "Why not use fastjson". Friends who are interested might as well take a look. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn "Why not use fastjson"!

Why give up fastjson?

The reason is that fastjson vulnerabilities occur frequently, which leads to the need to frequently urge various business lines to upgrade the fastjson version within the company to prevent security problems.

Fastjson frequently exposed security vulnerabilities in 2020 that could bypass the autoType switch to deserialize remote code execution and gain access to the server.

From the release of v1.2.59 in July 2019 to the release of v1.2.71 in June 2020, each version has an upgrade on AutoType, involving 13 official versions.

Version history related to AutoType in fastjson:

1.2.59 release to enhance the security fastjson when AutoType is opened

1.2.60 release, added AutoType blacklist, fixed denial of service security problem fastjson

1.2.61 release, adding AutoType security blacklist fastjson

1.2.62 release, adding AutoType blacklist, enhanced date deserialization and JSONPath fastjson

1.2.66 release, Bug repair security reinforcement, and do security reinforcement, supplement the AutoType blacklist fastjson

1.2.67 release, Bug repair security reinforcement, added AutoType blacklist fastjson

1.2.68 released, supports GEOJSON, and complements the AutoType blacklist

1.2.69 released to fix the newly discovered high-risk AutoType switch to bypass security vulnerabilities and supplement the AutoType blacklist

1.2.70 release to improve compatibility and supplement the AutoType blacklist

1.2.71 release, supplement safety blacklist, no new use, preventive supplement

In contrast, other json frameworks, such as Gson and Jackson, have fewer vulnerabilities and fewer high-risk vulnerabilities, which is the main reason why companies want to replace the framework.

Fastjson alternative

This paper mainly discusses the actual combat problem of Gson replacing fastjson framework, so we do not discuss the advantages and disadvantages of various json frameworks in detail here, only give the conclusion.

After evaluation, two main json frameworks, Jackson and Gson, are taken into consideration and compared with fastjson.

Characteristics of three json frameworks FastJson

High speed

Fastjson is faster than other JSON libraries, and its performance has never been surpassed by other Java-implemented JSON libraries since fastjson released version 1.1.x in 2011.

Widely used

Fastjson is widely used in Alibaba and deployed on tens of thousands of servers. Fastjson is widely accepted in the industry. It was selected as one of the most popular domestic open source software by Open Source China in 2012.

The test is complete

Fastjson has a lot of testcase, and in version 1.2.11, there are more than 3321 testcase. Regression testing will be carried out for each release to ensure stable quality.

Easy to use

Fastjson's API is very concise.

Jackson

Easy to use-jackson API provides a high-level look and feel to simplify common use cases.

No need to create mappings-API provides default mapping for most object serialization.

High performance-fast, low memory footprint, suitable for large object charts or systems.

Clean JSON-jackson creates a clean and compact JSON result, which is easy to read.

No dependency-the library does not require any other libraries except JDK.

Gson

Provides a mechanism that makes converting Java objects to JSON or vice versa as simple as using toString () and constructors (factory methods).

Allows pre-existing immutable objects to be converted to JSON or vice versa.

Allow the representation of custom objects

Support arbitrarily complex objects

Output lightweight and easy-to-read JSON

Performance comparison

The source code of performance comparison written by colleagues:

Https://github.com/zysrxx/json-comparison

This article does not discuss the performance differences in detail, after all, it involves a lot of implementation ideas and optimization of various frameworks, so only the conclusions are given:

1. Serialized single object performance Fastjson > Jackson > Gson, in which the performance gap between Fastjson and Jackson is very small, and the performance of Gson is poor.

two。 The performance of serializing large objects is Jackson > Fastjson > Gson. When serializing large Json objects, Jackson > Gson > Fastjson,Jackson has obvious performance advantages when serializing big data

3. Deserialization single object performance Fastjson > Jackson > Gson, the performance gap is small

4. Deserialization of large object performance Fastjson > Jackson > Gson, the performance gap is very small

The final solution Jackson is suitable for high-performance scenarios, and Gson is suitable for high-security scenarios. For new project repositories, fastjson is no longer used. For the stock system, considering the replacement cost of Json, the following options are available: if the project does not use the autoType function, it is recommended to switch directly to non-fastjson. If the switching cost is high, you can consider continuing to use fastjson and turn off safemode. The autoType function is used in the business, and it is recommended to promote obsolete fastjson. Replace dependency considerations

The characteristics of enterprise projects or large-scale projects:

The code structure is complex and maintained by many people in the team. Undertake important online business, in the event of serious bug will lead to major accidents. If it is an old project, it may be lack of documentation and cannot be modified at will. The project has many development branches and is constantly iterating online.

So for large projects, it is complex and painful to migrate the underlying fastjson to gson, as well as for other dependent replacements.

I have summarized the following issues that should be paid special attention to in the process of replacing project dependencies.

Be careful, be careful, be careful.

You can't be too careful. If the project you want to change is a very important business, it can be very costly if you make a mistake. And, for the business side and the product team, there are no new features online, but the system explodes, which is an "unbearable" thing. Although you may feel aggrieved, only you or your team knows that although the business does not seem to have changed, earth-shaking changes have taken place at the bottom of the code.

So, be careful!

Make a good communication between the development team and the test team

In the process of relying on replacement, we need to make a good project planning, such as sub-module replacement, strict subdivision of scheduling.

Do a good job of pre-planning, development and testing can work in an orderly way.

Between developers, it is necessary to communicate development considerations in advance, such as relying on version issues to prevent multiple developers from modifying the code at the same time, and it is embarrassing to find that different versions are used and different interface usage is different. and it takes extra time to deal with.

As for testing, it is more important to communicate well in advance. In general, testing does not pay much attention to this kind of technical project that does not change for the business, because it is neither optimization speed nor new functionality. But in fact, the migration involves the bottom layer, it is easy to appear BUG. It takes a lot of testing time to make the test team understand that the replacement project is dependent, and the cost is no less than the new features. Let them pay as much attention as possible.

Do well the regression / interface test

It is mentioned above that the test team needs to invest a lot of man-hours, which are mainly spent on the overall regression of the project function, that is, regression testing.

Of course, it's not just business regression testing, but interface regression testing if possible.

If the company has an interface management platform, it can greatly improve the efficiency of this kind of project testing.

For example, after the modification of a module is completed, deploy an online version in the test environment (or sandbox environment), deploy a modified version, and directly compare the data returned by the API. Generally speaking, it is Json comparison, and there are many Json comparison tools on the Internet:

Https://www.sojson.com/

Consider the performance differences before and after migration

As the performance comparison between Gson and Fastjson described above, the replacement framework needs to pay attention to the performance differences between frameworks, especially for traffic services, that is, high concurrency projects, if the response time changes greatly, it will attract the attention of upstream and downstream, leading to some additional consequences.

Replace Fastjson with Gson

Here we summarize two methods commonly used in json framework, and post detailed code examples to help you get started with Gson quickly and switch seamlessly!

Json deserialization String jsonCase = "[{\" id\ ": 10001,\" date\ ": 1609316794600,\" name\ ":\" Xiaoming\ "}, {\" id\ ": 10002,\" date\ ": 1609316794600,\" name\ ":\" Xiao Li\ "]"

/ / fastjson

JSONArray jsonArray = JSON.parseArray (jsonCase)

System.out.println (jsonArray)

System.out.println (jsonArray.getJSONObject (0) .getString ("name"))

System.out.println (jsonArray.getJSONObject (1). GetString ("name")

/ / output:

/ [{"date": 1609316794600, "name": "Xiaoming", "id": 10001}, {"date": 1609316794600, "name": "Xiao Li", "id": 10002}]

/ / Xiaoming

/ / Xiao Li

/ / Gson

JsonArray jsonArrayGson = gson.fromJson (jsonCase, JsonArray.class)

System.out.println (jsonArrayGson)

System.out.println (jsonArrayGson.get (0). GetAsJsonObject (). Get ("name"). GetAsString ()

System.out.println (jsonArrayGson.get (1). GetAsJsonObject (). Get ("name"). GetAsString ()

/ / output:

/ [{"id": 10001, "date": 1609316794600, "name": "Xiaoming"}, {"id": 10002, "date": 1609316794600, "name": "Xiao Li"}]

/ / Xiaoming

/ / Xiao Li

It can be seen that the difference between the two is mainly in the various types of get, the method of calling Gson has changed, but not much.

So, let's see if an exception occurs in the deserialization of empty objects:

String jsonObjectEmptyCase = "{}"

/ / fastjson

JSONObject jsonObjectEmpty = JSON.parseObject (jsonObjectEmptyCase)

System.out.println (jsonObjectEmpty)

System.out.println (jsonObjectEmpty.size ())

/ / output:

/ / {}

/ / 0

/ / Gson

JsonObject jsonObjectGsonEmpty = gson.fromJson (jsonObjectEmptyCase, JsonObject.class)

System.out.println (jsonObjectGsonEmpty)

System.out.println (jsonObjectGsonEmpty.size ())

/ / output:

/ / {}

/ / 0

Nothing unusual, happy.

Look at the empty array. After all, [] feels more error-prone than {}.

String jsonArrayEmptyCase = "[]"

/ / fastjson

JSONArray jsonArrayEmpty = JSON.parseArray (jsonArrayEmptyCase)

System.out.println (jsonArrayEmpty)

System.out.println (jsonArrayEmpty.size ())

/ / output:

/ / []

/ / 0

/ / Gson

JsonArray jsonArrayGsonEmpty = gson.fromJson (jsonArrayEmptyCase, JsonArray.class)

System.out.println (jsonArrayGsonEmpty)

System.out.println (jsonArrayGsonEmpty.size ())

/ / output:

/ / []

/ / 0

There is nothing wrong with both frameworks, and the analysis is perfect.

Paradigm treatment

Parsing generics is a very common feature, and most of the fastjson code in our project is parsing json and Java Bean.

/ / entity class

User user = new User ()

User.setId (1L)

User.setUserName ("Jack Ma")

/ / fastjson

List userListResultFastjson = JSONArray.parseArray (JSON.toJSONString (userList), User.class)

List userListResultFastjson2 = JSON.parseObject (JSON.toJSONString (userList), new TypeReference () {})

System.out.println (userListResultFastjson)

System.out.println ("userListResultFastjson2" + userListResultFastjson2)

/ / output:

/ / userListResultFastjson [User [Hash = 483422889, id=1, userName= Jack Ma], null]

/ / userListResultFastjson2 [User [Hash = 488970385, id=1, userName= Jack Ma], null]

/ / Gson

List userListResultTrue = gson.fromJson (gson.toJson (userList), new TypeToken () {} .getType ())

System.out.println ("userListResultGson" + userListResultGson)

/ / output:

/ / userListResultGson [User [Hash = 1435804085, id=1, userName= Jack Ma], null]

As you can see, Gson also supports generics.

List/Map write

Fastjson differs from Gson in that Gson does not support writing List directly to value, while fastjson does.

Therefore, Gson can only parse the List and write it to value. For more information, please see the following code:

/ / entity class

User user = new User ()

User.setId (1L)

User.setUserName ("Jack Ma")

/ / fastjson

JSONObject jsonObject1 = new JSONObject ()

JsonObject1.put ("user", user)

JsonObject1.put ("userList", userList)

System.out.println (jsonObject1)

/ / output:

/ / {"userList": [{"id": 1, "userName": "Jack Ma"}, null], "user": {"id": 1, "userName": "Jack Ma"}}

/ / Gson

JsonObject jsonObject = new JsonObject ()

JsonObject.add ("user", gson.toJsonTree (user))

System.out.println (jsonObject)

/ / output:

/ / {"user": {"id": 1, "userName": "Jack Ma"}, "userList": [{"id": 1, "userName": "Jack Ma"}, null]}

In this way, Gson looks less convenient than fastjson, because the List is placed in the form of gson.toJsonTree (user). In this way, you can't enter the object first and modify it later. (some students are used to putting objects in first and then modifying objects, so the code has to be changed.)

Conversion between hump and underline

The hump conversion underline depends on modifying the serialization mode of Gson to LOWER_CASE_WITH_UNDERSCORES.

GsonBuilder gsonBuilder = new GsonBuilder ()

GsonBuilder.setFieldNamingPolicy (FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES)

Gson gsonUnderScore = gsonBuilder.create ()

System.out.println (gsonUnderScore.toJson (user))

/ / output:

/ / {"id": 1, "user_name": "Jack Ma"}

Frequently asked questions demining

The following is a list of the pits we stepped on during the Gson migration of the company's project. these pits now feel untechnical in writing. But that's why I wrote this article to help you avoid these hard-to-find potholes.

Some of these problems are found during regression testing, some are found during self-testing, and some are found after being online, such as problems that will not be detected when Swagger is failed.

Date serialization is different.

I don't know if you have thought about a question, if you have a cache system in your project, use fastjson write cache, after you switch Gson, you need to parse it with Gson. So it is important to make sure that the parsing logic of the two frameworks is the same, but it is clear that this wish is good.

During the testing process, the Date type was found, and parsing is different in the two frameworks.

Fastjson:Date is parsed directly to UnixGson: directly serialized to the standard format Date

As a result, when Gson deserializes the json, it directly reports an error and cannot be converted to Date.

Solution:

Create a new class dedicated to parsing Date types:

Import com.google.gson.TypeAdapter

Import com.google.gson.stream.JsonReader

Import com.google.gson.stream.JsonWriter

Import java.io.IOException

Import java.util.Date

Public class MyDateTypeAdapter extends TypeAdapter {

@ Override

Public void write (JsonWriter out, Date value) throws IOException {

If (value = = null) {

Out.nullValue ()

} else {

Out.value (value.getTime ())

}

}

@ Override

Public Date read (JsonReader in) throws IOException {

If (in! = null) {

Return new Date (in.nextLong ())

} else {

Return null

}

}

}

Next, when you create a Gson, you put it into a special processing class for Date:

Gson gson = new GsonBuilder () .registerTypeAdapter (Date.class,new MyDateTypeAdapter ()) .create ()

This allows Gson to process Date as Unix.

Of course, this is only for compatibility with the old cache, and if you feel that your repository does not have such concerns, you can ignore this issue.

SpringBoot exception

After switching to Gson, the interface of the Web project built with SpringBoot cannot be directly requested. The error report is similar to:

Org.springframework.http.converter.HttpMessageNotWritableException

Because the default Mapper for SpringBoot is Jackson parsing, after we switch to Gson as the return object, Jackson cannot parse.

Solution:

Add to application.properties:

# Preferred JSON mapper to use for HTTP message conversion

Spring.mvc.converters.preferred-json-mapper=gson

Swagger exception

This problem is similar to the SpringBoot exception above, because Gson was introduced into SpringBoot, causing swagger to fail to parse json.

Adopt a solution similar to the following (add Gson adapters):

Http://yuyublog.top/2018/09/03/springboot%E5%BC%95%E5%85%A5swagger/

GsonSwaggerConfig.java@Configuration

Public class GsonSwaggerConfig {

/ / set swagger to support gson

@ Bean

Public IGsonHttpMessageConverter IGsonHttpMessageConverter () {

Return new IGsonHttpMessageConverter ()

}

}

IGsonHttpMessageConverter.javapublic class IGsonHttpMessageConverter extends GsonHttpMessageConverter {

Public IGsonHttpMessageConverter () {

/ / Custom Gson Adapter

Super.setGson (new GsonBuilder ()

.registerTypeAdapter (Json.class, new SpringfoxJsonToGsonAdapter ())

.serializeNulls () / / Null values also participate in serialization

.create ())

}

}

SpringfoxJsonToGsonAdapter.javapublic class SpringfoxJsonToGsonAdapter implements JsonSerializer {

@ Override

Public JsonElement serialize (Json json, Type type, JsonSerializationContext jsonSerializationContext) {

Return new JsonParser () .parse (json.value ()

}

}

@ Mapping JsonObject as input parameter exception

Sometimes, we use something like this in the input parameter:

Public ResponseResult submitAudit (@ RequestBody JsonObject jsonObject) {}

If you use this code, you are actually using Gson to parse json strings. But the risk of this way of writing is very high, usually please try to avoid using JsonObject to accept parameters directly.

In Gson, if JsonObject has a numeric field, it will be uniformly serialized to double, that is, count = 0 will be serialized to count = 0.0.

Why is this happening? To put it simply, when Gson parses json to Object types, it defaults to using double conversion for numeric types.

If the Json corresponds to the Object type, it will eventually be resolved to the Map type; where the Object type is related to the specific value in the Json, such as the "" value of double quotes translated into STRING. We can see that numeric types (NUMBER) are all converted to Double types, so there is our previous problem that integer data is translated into Double types, such as 30 to 30.

Take a look at Gson's ObjectTypeAdaptor class, which inherits Gson's TypeAdaptor abstract class:

Solution:

The first plan: receive the input entity class, do not use JsonObject

The second solution: similar to the above solution to the Date type problem, define your own Adaptor to accept numbers and process them. I think this idea is feasible but difficult, and it may affect other types of parsing, which requires special attention when designing adapters.

At this point, I believe you have a deeper understanding of "Why not use fastjson". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.