Shulou(Shulou.com)05/31 Report--

Spark SQL how to operate the JSON field, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, I hope you can gain something.

Get_json_object

The first is get_json_object, which is used as follows:

Select get_json_object ('{"k": "foo", "v": 1. 0}','$.k') as k

You need to give get_json_object a json field name (or string), and then get the specific value in a jsonPath-like way.

This method is actually a bit troublesome, if I want to extract a field inside, I have to write something similar, very complicated.

From_json

The specific usage is as follows:

Select a.k from (select from_json ('{"k": "foo", "v": 1.0}','k STRING, v STRING',map ("", ")) as a)

This method can define a Schema for json, so that A.K can be used directly when using it, which is much simpler.

To_json

This method can convert the corresponding field into a json string, such as:

Select to_json (struct (*)) AS value

You can convert all fields to json strings, then represent them as value fields, and then you can write the value field to Kafka. Isn't it easy.

Working with JSON datasets with a large number of fields

JSON data is usually semi-structured and unfixed. In the future, we will extend Spark SQL support for JSON to handle situations where each object in the dataset may have a quite different structure. For example, consider using the JSON field to save a dataset that represents a key / value pair of HTTP headers. Each record may introduce a new header type, and using a different column for each record will result in a very wide pattern. We plan to support automatic detection of this situation, using the map type instead. Therefore, each row can contain Map so that its key / value pairs can be queried. In this way, Spark SQL will handle less structured JSON datasets, pushing the boundaries of the kind of queries that SQL-based systems can handle.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.