Shulou(Shulou.com)05/31 Report--

This article mainly introduces the example analysis of MDLog, which is very detailed and has certain reference value. Friends who are interested must finish it!

ReplayThread class: responsible for replay event handling for log.

RecoveryThread class: responsible for recovery event handling for log.

SubmitThread class: responsible for submit event handling for log.

Map segments; records the sequence of log

Set expiring_segments; records the log collection of expiring

Set expired_segments; records the log collection of expired

Uint64_t event_seq; records the current sequence value of log event

Int expiring_events; records the number of log of expiring

Int expired_events; records the number of log of expired

MDLog::write_head ()

| | _ _ Journaler::write_head () directly calls the corresponding function of Journaler class for processing |

MDLog::get_read_pos ()

| | _ Journaler::get_read_pos () |

MDLog::get_write_pos ()

| | _ Journaler::get_write_pos () |

MDLog::get_safe_pos ()

| | _ Journaler::get_write_safe_pos () |

MDLog::create ()

| | _ _ create C_GatherBuilder class object |

| | _ _ sets the finisher function of C_GatherBuilder class |

| | _ _ determine the default journal number of inode, that is: ino = MDS_INO_LOG_OFFSET+mds- > get_nodeid () |

| | _ _ create a Journaler class object, which is written to metadata pool |

| | _ _ sets the write-out error handling function of Journaler class objects |

| | _ _ sets Journaler class object writable |

| | _ Journaler::create () |

| | _ Journaler::write_head () |

| | _ _ create a JournalPointer class object, which is written to metadata pool |

| | _ _ set the front value of JournalPointer to inode number |

| | _ _ set the back of JournalPointer to 0 |

| | _ JournalPointer::save () |

| | _ C_GatherBuilder::activate () |

| | _ _ SubmitThread::create () creates a SubmitThread thread |

MDLog::open ()

| | _ _ RecoveryThread::set_completion () sets the completion callback function of RecoveryThread |

| | _ _ RecoveryThread::create () creates a RecoveryThread thread |

| | _ _ SubmitThread::create () creates a SubmitThread thread |

MDLog::reopen ()

| | _ _ Delete Journaler class objects |

| | _ _ RecoveryThread::join () waits for RecoveryThread to finish |

| | _ _ RecoveryThread::set_completion () sets the callback function MDLog::append () at the end of RecoveryThrad |

| | _ _ RecoveryThread::create () creates a RecoveryThread thread |

MDLog::append ()

| | _ _ Journaler::set_read_pos (Journaler::get_write_pos ()) sets the read/write position of the Journaler object pointing to the same place |

| | _ _ Journaler::set_expire_pos (Journaler::get_write_pos ()) sets the expire position of Journaler objects to write position |

| | _ _ Journaler::set_writable () sets Journaler to writable |

MDLog::_start_entry ()

| | _ _ set cur_event=e and set cur_event to the current LogEvent to be processed |

| | _ _ event_seq++ increases the serial number of Event |

| | _ _ get the EMetaBlob object from the LogEvent class object, namely: LogEvent::get_metablob () |

| | _ _ sets the event_seq of EMetaBlob to the current event_seq,last_subtree_map and the last segment |

MDLog::cancel_entry ()

| | _ _ set cur_event=NULL to clear cur_event |

| | _ _ Delete LogEvent parameter class object |

MDLog::_submit_entry ()

| | _ _ set cur_event = NULL |

| | _ _ get the last LogSegment object from the segments collection |

| | _ _ increase the number of event of LogSement class objects, namely: num_events++ |

| | _ _ sets LogEvent::_segment=LogSement class object |

| | _ _ LogEvent::update_segment () updates the segment field of LogEvent |

| | _ _ update the pending_events collection with LogEvent as a parameter |

| | _ num_events++ increases the number of events |

| | _ unflushed++ increases the number of unflushed |

| | _ _ if the type of LogEvent is EVENT_SUBTREEMAP or EVENT_IMPORTFINISH and mds is in resolve status |

| | _ _ exit directly |

| | _ _ if LogSegment:: end/Journaler::get_layout_period ()! = LogSegment::offset/Journaler::get_layout_period () |

| | _ _ start_new_segment () |

| | _ _ if the LogEvent type is EVENT_SUBTREEMAP_TEST |

| | _ _ create LogEvent class object |

| | _ _ sets the type of LogEvent class object to EVENT_SUBTREEMAP_TEST |

| | _ _ submit_entry () |

MDLog::_submit_thread ()

| | _ _ determine whether the current MDS process is stopping, and if so, exit directly |

| | _ _ determine whether mds_log_pause is true. If so, call submit_cond.Wait () |

| | _ _ determines whether the pending_events collection is empty. If so, call submit_cond.Wait () |

| | _ _ get PendingEvent class object from pending_event |

| | _ _ if PendingEvent contains a valid LogEvent |

| | _ _ get LogEvent and LogSegment |

| | _ _ encode LogEvent to bufferlist |

| | _ _ set LogEvent::set_start_off (write_pos) set the start offset value of LogEvent to write position value |

| | _ _ Journaler::append_entry (bufferlist) writes encoded LogEvent to Journaler |

| | _ _ update LogSegment::end to the latest write position |

| | _ _ Journaler::wait_for_flush () waits for Journaler flush LogEvent to be sent to disk. After completing the flush, call the callback function to set the latest write position of MDLog::safe_pos= |

| | _ Journaler::flush () |

| | _ _ Delete LogEvent class objects |

| | _ _ if PendingEvent does not contain a valid LogEvent |

| | _ _ Journaler::wait_for_flush () waits for Journaler flush LogEvent to be sent to disk. After completing flush, call the callback function to set MDLog::safe_pos=Journaler::get_write_pos () |

| | _ Journaler::flush () |

MDLog::wait_for_safe () if there is still an event of pending, do not do wait_for_flush

| | _ _ determines whether pending_events is not empty |

| | _ _ add a NULL PendingEvent class object to the end of the pending_events collection |

| | _ _ set no_pending=false |

| | _ submit_cond.Signal () |

| | _ _ judge no_pending==true |

| | _ Journaler::wait_for_flush () |

MDLog::flush () if there is still an event of pending, do not do flush

| | _ _ determines that the pending_events collection is not empty |

| | _ _ insert a NULL PendingEvent class object into the end of the pending_event collection |

| | _ _ set do_flush=false |

| | _ _ judge do_flush==true |

| | _ Journaler::flush () |

MDLog::shutdown ()

| | _ _ determine whether SubmitThread is running |

| | _ _ SubmitThread::join () waits for SubmitThread process to stop |

| | _ _ Journaler::shutdown () calls Journaler's shutdown () function |

| | _ _ determine whether ReplayThread is running |

| | _ _ ReplayThread::join () waits for ReplayThread process to stop |

| | _ _ determine whether RecoveryThread is running |

| | _ _ RecoveryThread::join () waits for RecoveryThread process to stop |

MDLog::_prepare_new_segment ()

| | _ _ get the value of seq as event _ seq + 1 |

| | _ _ create a new LogSegment class object at segments [seq] |

| | _ MDCache::advance_stray () |

MDLog::_journal_segment_subtree_map ()

| | _ _ MDCache::create_subtree_map () gets an ESubtreeMap from MDCache |

| | _ _ set the event_ seq value of ESubtreeMap to the last element of segment |

| | _ _ submit_entry () |

MDLog::_start_new_segment ()

| | _ _ prepare_new_segment () |

| | _ _ journal_segment_subtree_map () |

MDLog::trim ()

| | _ _ get max_segments and max_events from configuration file |

| | _ _ traverse the segments collection |

| | _ _ if the LogSegment is in the pending_events collection, it exits directly and cannot perform trim operation |

| | _ _ if the LogSegment is in the expiring_segments or expired_segments collection, traverse the next one |

| | _ _ insert the LogSegment into the expiring_segments collection |

| | _ try_expire () |

| | _ _ trim_expired_segments () |

The MDLog::trim_all () process is similar to MDLog::trim (), except that there is no limit on the number of trim to traverse all the segments.

MDLog::try_expire ()

| | _ _ removes the specified LogSegment from the expiring_segments collection |

| | _ _ expired () |

| | _ _ insert the LogSegment into the expired_segments collection |

MDLog::_trim_expired_segments ()

| | _ _ traverse the segments collection |

| | _ _ determines whether LogSegment is in the expired_segments collection, and stops traversing if it is not. |

| | _ _ remove LogSegment from the expired_segments collection |

| | _ _ remove LogSegment from the segments collection |

| | _ _ Delete LogSegment |

| | _ _ set trim=true |

| | _ if trim==true |

| | _ Journaler::write_head (0) |

MDLog::_maybe_expired ()

| | _ try_expire () |

MDLog::replay ()

| | _ _ determine whether the read pos of Journaler is consistent with that of write pos |

| | _ _ does not need replay and returns directly |

| | _ _ waitfor_replay.push_back (c) add replay waiter |

| | _ _ ReplayThread::create () creates a new ReplayThread |

MDLog::_replay_thread () a separate thread that performs the replay operation

| | _ _ determine whether there is any unread data in Journaler |

| | _ _ Journaler::wait_for_readable () waits for Journaler to be readable and has no unread data |

| | _ _ determine whether Journaler has an error message |

| | _ _ handle the error message of Journaler |

| | _ _ if the Journaler is unreadable and the read pos==write pos of Journaler |

| | _ _ exit directly |

| | _ _ get the read position of Journaler |

| | _ _ Journaler::try_read_entry () reads data from Journaler class objects |

| | _ _ get LogEvent class object |

| | _ _ set the start offset of LogEvent to read position |

| | _ _ LogEvent is EVENT_SUBTREEMAP or EVENT_RESETJOURNAL |

| | _ _ create a new LogSegment class object in the segments collection |

| | _ _ if segments collection is not empty |

| | _ _ set LogEvent::_segment=get_current_segment () |

| | _ _ set LogEvent::_segment- > end = Journaler::get_read_pos () |

| | _ LogEvent::replay () |

| | _ _ set safe_pos = Journaler::get_write_safe_pos () |

MDLog::_recovery_thread ()

| | _ _ create JournalPointer class object |

| | _ _ JournalPointer::load (mds- > objecter) load data from JournalPointer |

| | _ _ if JournalPointer.back is not empty, it means that writing Journal is not completed |

| | _ _ create Journaler class object based on JournalPointer.back |

| | _ _ Journaler::recover () calls the recover () function of the Journaler class object for recover operation |

| | _ _ create a Journaler class object from JournalPointer.front |

| | _ _ Journaler::recover () calls the recover () function of the Journaler class object for recover operation |

| | _ _ if MDS is in standby replay mode or stream_format () > = mds_journal_format |

| | _ _ Journaler::set_write_error_handler () sets the write error handler of Journaler |

| | _ _ reformat_journal () |

MDLog::_reformat_journal ()

| | _ _ determine the backvalue of JournalPointer based on the front value of parameter JournalPointer |

| | _ _ create a new Journaler based on the back of parameter JournalPointer |

| | _ Journaler::set_writeable () |

| | _ Journaler::create () |

| | _ Journaler::write_head () |

| | _ _ read data from the parameter's old_journaler to bufferlist |

| | _ _ Journaler::append_entry (bufferlist) writes the data read by old_journaler to the new Journaler |

| | _ Journaler::flush () |

Add to your understanding of MDLog:

The core processing flow of MDLog::create () is as follows:

1. Get the inode number corresponding to Journaler, that is, MDS_INO_LOG_OFFSET+mds- > get_nodeid ()

2. Create a Journaler class object (Journaler is written to metadata pool)

3. Create a JournalPointer class object (with mds- > get_nodeid () and metadata pool as parameters)

4. Set the front of JournalPointer class object = inode,back = 0

5. Save the JournalPointer class object, that is, jp- > save (mds- > objecter, gather.new_sub ())

6. Create the SubmitThread thread class, that is, submit_thread.create (), and then execute the entry () function of the SubmitThread class, that is, _ submit_thread ()

The main processing flow of MDLog::_submit_thread () is as follows:

1. Traverse the pending_event array. If the array is empty or the PendingEvent in the array is empty, then traverse again

2. Get PendingEvent class objects from pending_event array.

3. Get LogEvent from the PendingEvent class object, and then get LogSegment from the LogEvent class object

4. For LogSegment, serialize its header information. The serialized header information is as follows:

EVENT_NEW_ENCODING

one

_ type

Mdsmap- > get_up_features ()

5. Get the location where the current Journaler class object is written to the log, that is, journaler- > get_write_pos ()

6. Write the serialized header information to journaler, that is, journaler- > append_entry (bl)

7. Flush journaler to disk, that is, journaler- > wait_for_flush ()

MDLog::_recovery_thread ()

1. Get the JournalPointer class object according to mds- > get_nodeid () and metadata pool

2. Read the object of JournalPointer, that is, jp.load (mds- > objecter)

3. If JournalPointer does not have object, create a front inode number of JournalPointer and save JournalPointer (jp.save (mds- > objecter))

4. If the jp.back is not empty, there is an error in writing back the log, so you need to delete the log corresponding to jp.back.

| | _ _ create Journaler class object based on jp.back |

| | _ _ execute Journaler::recover () |

| | _ _ execute Journaler::erase () |

| | _ _ update JournalPointer |

| | _ _ set jp.back=0 |

| | _ _ call jp.save (mds- > objecter) |

5. Create Journaler class objects according to jp.front

6. Execute Journaler::recover ()

7. Execute _ reformat_journal ()

MDLog::_reformat_journal () writes the log information to the back of JournalPointer. If the write is successful, set the front=back and back=0 of JournalPointer.

1. Get the backvalue of JournalPointer

2. Save the JournalPointer class object, that is, jp.save ()

3. Create Journaler class objects according to jp.back

4. Read the log information from the old log, that is, old_journal- > try_read_entry (bl)

5. Write the old log information to the new log, that is, new_journal- > append_entry (bl)

6. Refresh the log to disk, that is, new_journal- > flush ()

7. Change the front and backvalue of JournalPointer and save JournalPointer, that is, jp.save ()

8. Delete the old log, that is, old_journal- > erase ()

9. Update JournalPointer, that is, jp.back=0 and jp.save ()

MDLog::_start_entry ()

1. Set cur_event to the current entry to be processed

2. Increasing the value of event_seq

3. Update the event_ seq and last_subtree_ map values of LogEvent

MDLog::_submit_entry ()

1. Clear the cur_ eventvalue

2. Get the LogSegment from the end of the segments array

3. Set _ segment of LogEvent to LogSegment in segments array

4. Update the pending_events array, that is, pending_ events [ls-> seq] .push _ back (PendingEvent (le, c))

5. Update num_ events value and unflushed value

MDLog::_prepare_new_segment ()

1. According to the event_ seq value, the seq value is obtained, that is, seq = event_seq + 1

2. Update the segments array, namely: segments [seq] = new LogSegment (seq)

MDLog::trim ()

The two functions MDLog::trim_all () are used to trim the LogSegment that meets the condition from the segments array. Here you use two arrays, expiring_segments and expired_segments, to save the LogSegment in the trim process.

MDLog::replay () when the read_pos and write_pos of journaler are inconsistent, replay operation is required

1. Update the waitfor_replay array

2. Set already_replayed=true

3. Create the ReplayThread thread class, and then execute the ReplayThread.entry () function, that is, execute _ replay_thread ()

MDLog::_replay_thread ()

1. Read the log contents from Journaler, that is, journaler- > try_read_entry (bl)

2. Parse the LogEvent from the log, namely: LogEvent::decode (bl)

3. Update the segments array, that is, segments [event _ seq] = new LogSegment (event_seq, pos)

4. Update the _ segment of LogEvent

5. Execute the replay () operation of LogEvent

The above is all the content of this article "sample Analysis of MDLog". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.