Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "how to simplify Hadoop cloud deployment". In the operation process of actual cases, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to handle these situations! I hope you can read carefully and learn something!

In response to growing user interest in cloud services for big data management and analytics applications, vendors have begun to streamline cloud deployment processes for Hadoop and try to lower the purchase price of cloud Hadoop.

Big data and cloud computing have become important to Hadoop vendors and some big data technology companies. These companies are experimenting with new ways to simplify the steps and reduce deployment costs for users of Hadoop cloud systems.

Cloudera, for example, adds metering capabilities to its Cloudera Director tool to manage distributed clusters built into Hadoop. This allows Cloudera users to adopt a usage-based pricing model without having to pay per node, which allows them to run ad hoc systems built for specific purposes and free up resources after use, thus avoiding rising costs.

In a big data cloud environment with a single Cloudera Director instance, users can now deploy clusters across multiple regions within it. In addition, a new version of Cloudera Enterprise, a Hadoop-based big data platform, enables the Apache Impala SQL-on-Hadoop query engine to run directly on Amazon Simple Storage Service(S3) data stores. This enables query operations without moving data to the Hadoop distributed file system, which also facilitates the deployment of temporary systems on the AWS cloud.

On-demand pricing and support for Impala-on-S3 are seen as useful by Narasimhan Sampath, a systems architect at Choice Hotels International Inc., which runs Cloudera-based clusters on the AWS cloud that work in conjunction with technologies such as the Spark data processing engine and Kafka information query system to support a variety of self-service analytics applications.

Migrate your cluster to the cloud

At the Strata + Hadoop World 2016 conference, Sampath said Choice followed the BYOC approach and deployed its clusters to the cloud on demand. For example, a cluster in a marketing department can be deployed to the cloud, complete a job, and then free up resources. Similarly, the development team's cluster runs 12 hours a day and then shuts down at night to save the company's investment in the AWS cloud.

Cloudera's metered pricing approach is perfect for this scenario, Sampath said after the meeting. "I don't need to buy 500 [Cloudera] licenses unless I use them all the time. It's the same as Amazon's model. "

He added that Choice has worked very closely with Cloudera over the past six months, trying to connect S3 with Impala, which was originally released by Cloudera as open source software. Choice uses S3 as data storage. Sampath said Impala's support for new queries provides additional flexibility for BYOC strategies.

David Tishgart, Cloudera's director of cloud product marketing, says they've become more comfortable promoting cloud among customers. But until now, they didn't have a great solution for ad hoc systems, and they couldn't just add or subtract workloads. He admits that for this reason, most Cloudera users choose to run clusters in the cloud for the long term rather than take an ad hoc approach.

Catching up with the Hadoop Cloud

As more and more users demonstrate interest in the cloud, Clouder needs to compete with Amazon Elastic MapReduce(EMR), the Hadoop cloud platform provided by AWS. Cloudera also finds itself at a disadvantage in competing with Microsoft Azure HDInsight big data cloud service, which is based on Hortonworks Inc . distributed environment of Hadoop.

According to Gartner analyst Merv Adrian, EMR has made AWS the best Hadoop vendor in terms of user numbers. AWS initially lagged behind other Apache Hadoop competitors, but that changed two years ago and now has more users than all other vendors combined.

Hortonworks is also focused on extending Hadoop cloud capabilities, saying HDInsight now runs version 2.5 of the Hortonworks Data Platform (HDP). Hortonworks also supports integration with Microsoft's Azure Active Directory service and Apache Ranger. Apache Ranger is a framework for managing Hadoop data security and user access.

Although closely related to Microsoft Cloud environments, Hortonworks also offers a technical preview of HDP for AWS users to build temporary clusters using Spark and Apache Hive. "We understand workloads across all cloud environments," said Matt Morgan, senior vice president of global marketing for the company.

Paxata is also starting to use cloud environments. The vendor of self-service data preparation software offers a new tool called Paxata Connect, which aggregates data running on different Hadoop clusters, including data from independent cloud platforms. Nenshad Bardoliwalla, product officer at Paxata***, said that many Hadoop workloads have migrated to the cloud, and the temptation to create "ad hoc" clusters, run specific jobs, and then release resources is huge.

"How to simplify Hadoop cloud deployment" content introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.