Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to simplify Hadoop cloud deployment". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's ideas to study and learn "how to simplify Hadoop cloud deployment".

In response to the growing user awareness of big data's management and analysis of application cloud services, vendors have begun to simplify the cloud deployment process of Hadoop and try to reduce the purchase price of cloud Hadoop.

Big data and cloud computing have become very important to Hadoop vendors and some big data technology companies. These companies are trying to use new methods to simplify the steps for users to deploy Hadoop cloud systems and reduce their deployment costs.

For example, Cloudera adds metering capabilities to its Cloudera Director tool to manage distributed clusters built in Hadoop. This allows Cloudera users to adopt a usage-based pricing model without having to pay on a node-by-node basis, which allows them to run temporary systems built for specific purposes that can release resources after use and avoid rising costs.

In the big data cloud environment of a single Cloudera Director instance, users can now deploy clusters in multiple regions. In addition, the new version of Cloudera Enterprise, a Hadoop-based big data platform, enables the Apache Impala SQL-on-Hadoop query engine to run directly on Amazon Simple Storage Service (S3) data stores. This enables query operations without moving data to the Hadoop distributed file system, which also facilitates the deployment of temporary systems on the AWS cloud.

On-demand pricing and support for Impala-on-S3 are very useful in the eyes of Narasimhan Sampath, who is Choice Hotels International Inc. The company's system architect, which runs Cloudera-based clusters on the AWS cloud, works with technologies such as the Spark data processing engine and the Kafka information query system to support a variety of self-help analysis applications.

Migrate your cluster to the cloud

In the Strata + Hadoop World 2016 meeting, Sampath said that Choice followed the BYOC approach of deploying its own cluster to the cloud environment on demand. For example, a cluster of marketing departments can be deployed to the cloud, complete a job, and then release resources. Similarly, the development team's cluster runs for 12 hours a day and then shuts down at night to save the company's investment in the AWS cloud.

Cloudera's metering pricing is perfect for this scenario, Sampath said after the meeting. "I don't need to buy 500,000 (Cloudera) licenses unless I use these resources all the time. It's the same as Amazon's model."

He added that Choice has worked very closely with Cloudera over the past six months, trying to connect S3 and Impala,Impala, which was originally released by Cloudera as open source software. Choice uses S3 as the data store. Sampath said that Impala's support for new queries provides additional flexibility to the BYOC strategy.

David Tishgart, Cloudera's director of cloud product marketing, says they are more and more willing to promote cloud among their customers. But until now, they do not have a good solution for temporary systems, nor can they casually increase or reduce workloads. He admits that for this reason, most Cloudera users choose to run the cluster in the cloud for a long time, rather than temporarily.

Catch up with the Hadoop cloud

As more and more users show interest in the cloud, Clouder needs to compete with Amazon Elastic MapReduce (EMR), the Hadoop cloud platform provided by AWS. In addition, Cloudera finds itself at a disadvantage in competing with Microsoft's Azure HDInsight big data cloud service, which is based on Hortonworks Inc. The distributed environment of Hadoop.

According to Merv Adrian, an analyst at Gartner, EMR has made AWS a Hadoop supplier in terms of the number of users. AWS initially lagged behind other Apache Hadoop competitors, but that changed two years ago, and now AWS Hadoop has more users than other vendors combined.

Hortonworks is also focused on extending Hadoop cloud capabilities, saying that HDInsight is now running version 2.5 of the Hortonworks data platform (HDP). In addition, Hortonworks now supports integration of Microsoft's Azure Active Directory service and Apache Ranger. (Apache Ranger is a framework for managing Hadoop data security and user access)

Despite its close relationship with Microsoft's cloud environment, Hortonworks also offers a technical preview of HDP so that AWS users can use Spark and Apache Hive to build temporary clusters. "We know the workloads on all cloud environments," said Matt Morgan, the company's senior vice president of global marketing.

Paxata is also starting to use cloud environments. The provider of self-service data preparation software offers a new tool called Paxata Connect, which aggregates data running on different Hadoop clusters, including data from separate cloud platforms. Nenshad Bardoliwalla, product officer at Paxata***, said that many Hadoop workloads have been migrated to the cloud, and the temptation to create "temporary" clusters, run specific tasks, and then release resources is great.

Thank you for reading, the above is the content of "how to simplify Hadoop cloud deployment". After the study of this article, I believe you have a deeper understanding of how to simplify Hadoop cloud deployment, and the specific usage needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.