Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

What is the distributed memory file system Tachyon

2025-01-24 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Servers >

Share

Shulou(Shulou.com)05/31 Report--

This article analyzes "what the distributed memory file system Tachyon is like". The content is detailed and easy to understand, the "distributed memory file system Tachyon is how" interested friends can follow the editor's ideas slowly in-depth to read, I hope that after reading can be helpful to everyone. Let's follow the editor to learn more about "what the distributed memory file system Tachyon is like".

Tachyon is a distributed memory file system that can access files stored in Tachyon at the speed of accessing memory in a cluster. Tachyon is a kind of middleware between the distributed file system at the bottom and various computing frameworks at the upper level. its main responsibility is to land the files that do not need to be landed in DFS to the distributed memory file system to achieve shared memory, so as to improve efficiency, reduce memory redundancy, reduce GC time and so on.

Tachyon architecture

The architecture of Tachyon is the traditional Master-Slave architecture, which is similar to Hadoop. WorkflowManager in TachyonMaster is a Master process. Because it is to prevent single point problems, HA is done through Zookeeper, and multiple Standby Master can be deployed. Slave is made up of Worker Daemon and Ramdisk. Ramdisk uses off heap memory. The direct communication protocol between Master and Worker is Thrift.

The following figure shows the architecture of Tachyon:

Fault Tolerant

What is the fault-tolerant mechanism of Tachyon?

Tachyon uses the concept of lineage, which we are already familiar with in Spark's RDD, by asynchronously Checkpoint to the underlying file system of Tachyon.

When we write a file to Tachyon, Tachyon asynchronously sends the file to checkpoint in the background to its underlying storage, such as HDFS,S3.. Etc...

An Edge algorithm is used here to determine the order of checkpoint.

A better strategy is to checkpoint a newly generated file each time a current checkpoint is completed. Of course, intermediate files like Hadoop,Hive, which need to be deleted, do not need checkpoint.

As shown in the following figure:

On the allocation strategy for resources when recalculating:

Currently, Tachyon supports two resource allocation strategies:

1. Priority resource allocation strategy

2. Allocation strategy of fair scheduling.

On the distributed memory file system Tachyon is how to share here, I hope that the above content can make you improve. If you want to learn more knowledge, please pay more attention to the editor's updates. Thank you for following the website!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 232

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Servers

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report