Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "how to use Python library to manage big data". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

BigQuery

Google BigQuery is a very popular enterprise repository made up of Google Cloud platform (GCP) and Bigtable. This cloud service can well handle data of all sizes and execute complex queries in seconds.

BigQuery is a RESTful web service that enables developers to interact with Google's cloud platform to analyze a large number of data sets. Take a look at another example below.

I wrote an article that explained how to connect to BigQuery and then began to get information about the tables and datasets with which to interact. In this case, the Medicare dataset is an open source dataset that anyone can access.

Another thing about BigQuery is that it runs on Bigtable. It is important to understand that the warehouse is not a transactional database. Therefore, it cannot be thought of as an online transaction processing (OLTP) database. It is specially designed for big data. So its work is consistent with the processing of PB-level data sets.

Redshift and Sometimes S3

Next up are Amazon's popular Redshift and S3. AmazonS3 is essentially a storage service for storing and retrieving large amounts of data from anywhere on the Internet. With this service, you only have to pay for the storage space actually used. On the other hand, Redshift is a well-managed data warehouse that can effectively deal with PB-level data. The service can be queried faster using SQL and BI tools.

Amazon Redshift and S3 deal with data as a powerful combination: large amounts of data can be uploaded to Redshift repositories using S3. This powerful tool is very convenient for developers when programming in Python.

This is a script that chooses to use the basic connection of psycopg2. I borrowed the Jaychoo code. However, this once again provides a quick guide to how to connect and get data from Redshift.

PySpark

Let's leave the world of data storage systems and explore tools that help us process data quickly. Apache Spark is a very popular open source framework that can perform large-scale distributed data processing. It can also be used for machine learning. The cluster computing framework mainly focuses on simplified analysis. It works with resilient distributed datasets (RDD) and allows users to handle the management resources of the Spark cluster.

It is often used in conjunction with other Apache products, such as HBase. Spark processes the data quickly and then stores it in a table set up on another data storage system.

Sometimes, installing PySpark can be a challenge because it requires dependencies. You can see that it runs on top of JVM, so it requires the underlying infrastructure of Java to run. However, in the era of the prevalence of Docker, it is more convenient to use PySpark for experiments.

Alibaba uses PySpark to personalize web pages and target ads-just like many other large data-driven organizations.

If you are interested in Python, welcome to join us [python Learning Exchange] and get learning materials and source code for free.

Kafka Python

Kafka is a distributed publish-subscribe messaging system that allows users to maintain message sources in replication and partitioning topics.

These topics are basically logs that receive data from the client and store it in the partition. Kafka Python is designed as the official Java client that integrates with the Python interface. It is best used with new agents and is backward compatible with all older versions. Programming with KafkaPython requires both reference consumer (KafkaConsumer) and reference producer (KafkaProducer).

In Kafka Python, these two aspects coexist. KafkaConsumer is basically an advanced message consumer and will be used as the official Java client.

It requires agents to support group API. KafkaProducer is an asynchronous message generator that operates very similar to the Java client. Producers can use it across threads without problems, while consumers need multithreading.

Pydoop

Let's solve the problem. Hadoop itself is not a data storage system. Hadoop actually has several components, including MapReduce and Hadoop distributed File system (HDFS). So, Pydoop is in this list, but you need to match Hadoop to other layers (such as Hive) to work with the data more easily.

Pydoop is a Hadoop-Python interface that allows you to interact with HDFSAPI and write MapReduce work in pure Python code.

The library allows developers to access important MapReduce features, such as RecordReader and Partitioner, without knowing Java.

For most data engineers, Pydoop itself may be a little too basic. Most of you will probably write ETLs in Airbow that runs on these systems. But at least it's good to have a general understanding of your work.

This is the end of the content of "how to use Python Library to manage big data". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.