Shulou(Shulou.com)06/01 Report--

First, problems

Often encounter such a problem when copying, the xlog file that needs to be copied can not be found. So when will the xlog file be deleted? How many xlog files will be deleted? Which xlog files need to be retained? This paper will interpret these problems in principle.

Second, principle

After each checkpoint, xlog files that are no longer needed will be deleted or recycled as needed.

1. First estimate the amount of xlog generated between the two checkpoint. Based on this amount, the largest log file number in the future will be calculated, and the files that are no longer needed will be reclaimed and renamed to the log file number that will be used in the future:

1.1 UpdateCheckPointDistanceEstimate estimates the number of logs generated before checkpoint:

If (CheckPointDistanceEstimate

< nbytes)//上次估算量比这次估算的小，则更新为这次的估算量 CheckPointDistanceEstimate = nbytes; else//否则，适当增加 CheckPointDistanceEstimate =(0.90 CheckPointDistanceEstimate + 0.10 (double) nbytes); 2、计算上一次checkpoint时，所在的文件段号_logSegNo： XLByteToSeg(PriorRedoPtr, _logSegNo); 3、计算需要保留的文件段号：从该段号_logSegNo开始的文件都不能被删除，之前的需要删除或回收：根据备机请求以及wal_keep_segments计算KeepLogSeg(recptr, &_logSegNo);

4. Traverse all the xlog files in the pg_wal directory and delete: RemoveOldXlogFiles

4.1Skip the timeline for comparison. If the files in the pg_wal directory are smaller than _ logSegNo, they are deleted or recycled. So what conditions will be recycled next time?

-- RemoveXlogFile

4.2 calculate the future maximum file segment number recycleSegNo renamed by the recycled file:

1) if this is the first checkpoint, the future maximum segment number recycleSegNo= current segment file number + 10

2) otherwise, call the function XLOGfileslop to calculate:

2.1 estimate the log location at the end of the next checkpoint:

Distance= (2.0+checkpoint_completion_target) CheckPointDistanceEstimate

Distance=1.1

RecycleSegNo = (XLogSegNo) ceil (double) PriorRedoPtr + distance) / XLOG_SEG_SIZE)

2.2 minSegNo = PriorRedoPtr / XLOG_SEG_SIZE + ConvertToXSegs (min_wal_size_mb)-1

MaxSegNo = PriorRedoPtr / XLOG_SEG_SIZE + ConvertToXSegs (max_wal_size_mb)-1

2.3 if (recycleSegNo

< minSegNo) recycleSegNo = minSegNo; if (recycleSegNo >

MaxSegNo)

RecycleSegNo = maxSegNo

4.3 if the current segment file number endlogSegNo

< recycleSegNo，则调用InstallXLogFileSegment进行回收： 1)在endlogSegNo和recycleSegNo之间找一个free slot num，即没有该段文件号的xlog文件 2)将需要删除的文件名命名为该free slot号的文件名 3)如果没有找到free slot则直接删除该文件 --RemoveXlogFile 三、代码流程 1、checkpoint顶层函数CreateCheckPoint： CreateCheckPoint： XLogCtlInsert *Insert = &XLogCtl->

Insert;// identifies the insertion location curInsert = XLogBytePosToRecPtr (Insert- > CurrBytePos); / / position after adding header size / / (curInsert)% XLOG_BLCKSZ = = 0)? 0: (XLOG_BLCKSZ-(curInsert)% XLOG_BLCKSZ)) freespace = INSERT_FREESPACE (curInsert) / / whether there is free space if (freespace = = 0) {if (curInsert% XLogSegSize = = 0) {if (curInsert% XLogSegSize = = 0) on the page where the curInsert is located. If you are about to use the next segment file, skip the 36-byte curInsert + = SizeOfXLogLongPHD;//36 byte else//xlog segment file and skip the 20-byte curInsert + = SizeOfXLogShortPHD if you are about to use the next page. / / 20 byte} checkPoint.redo = curInsert;//xlog file, the actual position to be inserted RedoRecPtr = XLogCtl- > Insert.RedoRecPtr = checkPoint.redo;. / / the position at the end of the checkpoint record, that is, the location where the next xlog starts recptr = XLogInsert (RM_XLOG_ID,shutdown? XLOG_CHECKPOINT_SHUTDOWN: XLOG_CHECKPOINT_ONLINE);... PriorRedoPtr = ControlFile- > checkPointCopy.redo;// the starting position of the last checkpoint. If (PriorRedoPtr! = InvalidXLogRecPtr) {/ / from the start of the last checkpoint to the start of this checkpoint, the size of the generated XLOG is input parameter / * CheckPointDistanceEstimate: 1. When CheckPointDistanceEstimate=RedoRecPtr-PriorRedoPtr: 0.9% CheckPointDistanceEstimatekeeper 0.1 * (RedoRecPtr-PriorRedoPtr) * / UpdateCheckPointDistanceEstimate (RedoRecPtr-PriorRedoPtr); / / _ logSegNo = (PriorRedoPtr) / XLogSegSize XLByteToSeg (PriorRedoPtr, _ logSegNo); KeepLogSeg (recptr, & _ logSegNo) _ logSegNo--; RemoveOldXlogFiles (_ logSegNo, PriorRedoPtr, recptr)

2. Two macro definitions

# define UsableBytesInPage (XLOG_BLCKSZ-SizeOfXLogShortPHD) / / Note: not the first page of the file # define UsableBytesInSegment ((XLOG_SEG_SIZE / XLOG_BLCKSZ) * UsableBytesInPage-(SizeOfXLogLongPHD-SizeOfXLogShortPHD))

3. Function XLogBytePosToRecPtr

Static XLogRecPtrXLogBytePosToRecPtr (uint64 bytepos) {/ / bytepos: size of extra bytes occupied excluding headers of xlog pages fullsegs = bytepos / UsableBytesInSegment; bytesleft = bytepos% UsableBytesInSegment; / * 1, if bytesleft

< XLOG_BLCKSZ-32,则表示定位到第一页上，则文件偏移值跳过第一页页头大小 2、如果bytesleft >

= XLOG_BLCKSZ-32, it means that the location is not the first page * / if (bytesleft

< XLOG_BLCKSZ - SizeOfXLogLongPHD){ /* fits on first page of segment */ seg_offset = bytesleft + SizeOfXLogLongPHD; }else{ /* account for the first page on segment with long header */ seg_offset = XLOG_BLCKSZ;//先跳过第一页 bytesleft -= XLOG_BLCKSZ - SizeOfXLogLongPHD;//去掉第一页存放XLOG的大小 fullpages = bytesleft / UsableBytesInPage;//剩下的需要几个页 bytesleft = bytesleft % UsableBytesInPage;//剩下的偏移 // 文件偏移=第一页大小+剩下的几个页大小+剩下的偏移+最后一页的页头 seg_offset += fullpages * XLOG_BLCKSZ + bytesleft + SizeOfXLogShortPHD; } //result=(fullsegs) * XLOG_SEG_SIZE + seg_offset XLogSegNoOffsetToRecPtr(fullsegs, seg_offset, result); return result;} 4、函数KeepLogSeg static voidKeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo){ //segno为当前xlog即将插入位置在第几个文件上 XLByteToSeg(recptr, segno); //XLogCtl->

What is the minimum reserved value requested on the replicationSlotMinLSN; standby? Keep = XLogGetReplicationSlotMinimumLSN () / * compute limit for wal_keep_segments first * / if (wal_keep_segments > 0) {/ * first calculate the limits obtained by wal_keep_segments: 1. For example, if the wal_keep_ segments value is 10, if the file number segno of the current insert location is 5 Then push forward to 1 / 2, otherwise push forward the removable * / if before the segno after wal_keep_segments (segno 0 & & keep! = InvalidXLogRecPtr) {XLByteToSeg (keep, slotSegNo) If (slotSegNo

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.