Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "how to use GlusterFS Translator API". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Introduction to GlusterFS Translator API

Before going any further into translator api, we must understand two concepts about the context of this api.

One is the file system api, which exposes most file system functions mainly through the allocation table xlator_fops. Xlator_fops is a combination of linux vfs's file_operations, inode_operations, and and super_operations operations in an array. To understand how xlator works, you must understand what these calls do and what the relationships are, such as open/lookup/close,read/write,opendir/readdir,truncate,symlink, and so on.

One is the foundation of xlator api: it is asynchronous and callback-based. This means that the code that processes a request must be divided into two parts, one before entering the next xlator and the other after the next xlator processes the request. To put it another way, your allocation function (the first half of xlator_fops) calls the allocation function of the next xlator and returns directly without blocking. Your callback function (the second half) may be called quickly when you call the next xlator's allocation function, or at a later time from a completely different thread (often a network-transmitted Poll thread). In both cases, the callback cannot simply get the context from the stack. Glusterfs does provide a variety of ways to maintain and pass context between the allocation function and the callback function, but you can't just rely on the stack, you need to code something.

Allocation tables and default functions

The main allocation table for a xlator is often fops (the xlator dynamic library loading code uses the dlsym lookup operation op name). Fops contains pointers to all normal file system operations. Just need to populate the operation we are interested in in this xlator. Any other operation is populated at run time based on the default value, determines a callback function, and sends the request directly to the next xlator.

Default functions and callback functions serve other purposes in addition to providing default functionality. When we need to add a new function to a xlator, we can copy and modify the default functions for the same operation, and so on. This ensures a correct parameter list and reasonable default behavior. Each xlator may have another allocation table, such as the cbk table, which is used to manage the life cycle of inode and file descriptors (see the inode and file descriptor context sections)

STACK_WIND and STACK_ Unwin macros

STACK_WIND and STACK_UNWIND is implemented based on the callback mechanism. These two functions do not operate on a similar call stack in gdb, but rather an independently maintained stack, with each stack frame representing a call to xlators. When a fop is passed from the kernel to be called, the call is made as a request request, from FUSE xlator through DHT/AFR to client xlator to server xlator and finally to posix xlator. We can do whatever we need at the user space fop entry point (this should be fuse xlator), and then use STACK_WIND to pass the request along the xlator path set by the volume file.

Define STACK_WIND (frame, rfn, obj, fn, params...)

STACK_WIND (frame, default_setxattr_cbk, FIRST_CHILD (this), FIRST_CHILD (this) → fops → setxattr, loc, dict, flags, xdata)

The parameters of this function are explained as follows:

Frame: represents the stack frame stack frame of this request.

Rfn: the callback function to be called when the next xlator or other xlator completes the request (including the first half and the second half).

Obj: the xlator object to be passed, the next xlator, is not the xlator that invokes STACK_WIND. How to deal with the situation of multiple children?

Fn: the specific xlator operation to be invoked, from the fops allocation table of the next xlator.

Params... Any other entry point-related parameters (such as inodes, file descriptors, offsets, data buffers, etc.) for this fop, as mentioned above, the rfn callback function may be called within the STACK_WIND call, or later in another environment. We use STACK_UNWIND to complete the request without calling the next xlator, such as reading data from cache, or when the xlator layer finishes processing the req and sends the request back to the previous xlator from the callback at this layer. In fact, we'd better use STACK_UNWIND_STRICT. This function allows us to determine the type of request, such as open or close, and so on.

# define STACK_UNWIND_STRICT (op, frame, params …)

STACK_UNWIND_STRICT (symlink, frame, op_ret, op_errno, inode, buf, preparent, postparent, xdata)

Parameter interpretation

Parameter op: operation type, open/ or other. Used to check whether the extra parameters match this fop.

Params... Some additional parameters for this request

In addition, in op and params directly, there are two parameters op_ret and op_errno. Although it does not appear in the macro definition. But you still have to have it when you call it.

The current state of op_ret:fop (number of bytes read or written, often 0 indicates success of fop,-1 indicates failure)

Op_errno: standard error code, in the case of fop failure

Specific fop-related parameters have other special parameters, we can in params … To realize it in. But we often need these two parameters.

Frame: the currently requested stack frame.

This: represents the xlator object of this xlator instance

A callback function is similar to an allocation function, except that there is an additional parameter cookie between the two parameters. Is a pointer to an undefined data type, stored by the corresponding STACK_WIND. By default, this is a pointer to the stack frame generated by STACK_WIND, but STACK_WIND_COOKIE allows us to determine a different value. In this case, this extra parameter directly in rfn and obj can pass some context from the allocation function to its callback function. Note that this pointer cannot point to anything on the stack, because stack may no longer exist when the callback function is called.

Another thing to note: STACK_UNWIND may cause the entire call stack to exit, at which point the last STACK_UNWIND call will release all frames. Therefore, never expect the current frame content to be intact after calling STACK_UNWIND.

Per Request Context request context private variable

Each xlator stack frame has a local pointer, which is used to hold the xlator-related context. This is the most basic mechanism for saving the context between the allocation function and the callback. So we should get used to the following code pattern

/ * in dispatch function * /

Local = (my_locals_t *) GF_CALLOC (1Magnesiof (* local),...); if (! local) {/ * STACK_UNWIND with ENOMEM eror * /} / * fill in my_locals_t fields * / frame- > local = local;/* in callback * / local = frame- > local

Remember: each frame of frame's non-NULL local domain is released with GF_FREE when the stack is to be destroyed, without other cleanup work. If the local structure contains pointers or references to other objects, we need to clean up these resources ourselves. It is a good habit to clean up memory or other resources before destroying the stack, so you can't rely on GF_FREE to clean up automatically. The safest way is to define our own xlator-related destructor and invoke it manually before invoking STACK_UNWIND.

Inode and File Descriptor Context

Most allocation functions and callbacks take file descriptors or node inode as arguments. File descriptor (fd_t) or an inode (inode_t). Often, xlator needs to save some context about this xlator for some of these objects, so that information can be maintained during the full life cycle of a request. For example, DHT xlator needs to save the layout map of the directory and the known location of some inode. There is a set of functions to store the context for manipulating this type of context. In each function, the second argument is the pointer to the xlator object associated with this value. Value is 64-bit unsigned × ×.

Inode_ctx_put (inode, xlator, value) / * NB: put, not set * / put value in inode?

Inode_ctx_get (inode, xlator, & value) gets the value from inode to value.

Inode_ctx_del (inode, xlator, & value)

Fd_ctx_set (fd, xlator, value) / * NB: set, not put * /

Fd_ctx_get (fd, xlator, & value)

Fd_ctx_del (fd, xlator, & value)

The _ del function is actually a destructive get, returning first and then deleting the value. The inode function takes the form of two value, such as inode_ctx_put2, which operates on two values. Using xlator pointers as key/index is not just about looking good. When you want to delete an inode_t or fd_t, the delete code browses the context slot (presumably traversing all the xlator). For each context slot of xlator that uses inode_t or fd_t, look at the cbk table of xlator and call its forget or release. If the context is a pointer, you need to release the resource manually. The inode_t or fd_t pointer parameter passed to the allocation function and callback is just a borrowed reference. If you need to ensure that the object still exists in the future, you need to call inode_ref or fd_ref to achieve the permanent reference. You can inode_unref or fd_unref when references are no longer needed.

Inode and File Descriptor Context Dictionary and xlator options

Another common parameter type, dict _ t, is a general-purpose dictionary or hash-map data structure used to hold arbitrary values in the form of a string key/ index. For example, the value can be a variable size unsigned × ×, string, binary blob. String, binary blob may use the GLusterfs function free or glibc free. Or don't use free. The dict_t* and the * data_t objects that hold the value of the reference count, releasing the resource if the number of references is 0. As with inode and file descriptors, if you want to use an object that receives dict_t later, you need to use add _ ref and _ unref to manage its life cycle. Dictionaries are not just used for allocation functions and callback functions. It is also used to pass options to modules, such as options used by xlator's init function. In fact, the current init function body of xlator is mostly used to interpret the options contained in the dictionary. To add an option to xlator, you also need to add an entry to the options array of xlator. Each option can be a boolean/ integer / string / path / xlator name / other type. If it is a string, you can determine a list of correct values. The parsed options, along with other xlator-level information, is stored in the member variables of the private of the xlator_t structure structure (represented by this in most cases).

Child Enumeration and Fan Out subxlator enumeration and Fan out

Code in a xlator often needs to enumerate its child xlator. Either find a sub-xlator that meets the requirements or manipulate all sub-xlator. For example, for DHT xlator, you need to collect hash-layout maps from all sub-xlator to determine which sub-xlator; the file should be placed in. For AFR xlator, you need to extract the number of operations to be performed by the same file from the sub-xlator to determine the replication replication status. The code example is as follows: xlator_list_t * trav; xlator_t * xl

For (trav = this- > children; trav; trav = trav- > next) {xl = trav- > xlator; do_something (xl);}

If the goal is to fan out a request to all the sub-xlator, it takes some effort. The most common method in an allocation function is the following local → call_count = priv → num_children

* / STACK_UNWIND (frame,op_ret,op_errno,...);} return 0; in some cases, you can use STACK_WIND_COOKIE so that you can know which call returned in the callback. You can view the AFR. Call Frames and Call Stacks

All xlator functions use the asynchronous invocation specification of STREAMS like SEDA and AT&T. The required call frame and call stack data structure are almost functionally the same as those of the I requst packets O requst packets and irp stack in Windows NT. Many xlator functions are similar to those corresponding to fuse, but other parameters have been modified or added.

This is the most basic data structure in xlator api, and the call_frame_t and call_stack_txlators environment can be thought of as a thread library, with each request having its own lightweight thread. This thread has its own stack, but these call stacks are more structured than the c stack and often deviate from the strict FIFO semantics of the stack. Each call frame represents a function call, which may be nested. But these functions actually have two calls in c-the initial part executes and returns before any nested calls are made. The closing part is executed after the nested call is complete.

The members of call_frame_t are as follows:

Root: points to fake frames embedded in call_stack_t, unlike other separately assigned frame. It is the same for all frame in stack.

Parent points to the frame that calls this frame. To use c means to call the function calling function.

Next / prev points to the next / previous frame to be completed. Note that it is not the frame to be called

Local private data, the private data called for this frame. Is a local variable of the xlagor function, saved here, and can be maintained between the allocation function and the callback.

This points to the private global state of xlator.

Ret points to the callback function of the caller (upper xlator), which is executed when the req of this xlator is completed.

The ref_count and complete scheduler is used to track which frame is active, which is completed, and which needs to resume execution.

Lock and cookie:

Normally, after the new frame push arrives on the stack, next is the same as parent. However, a frame (frame) can branch. For example, the frame requested by stripe_writev generates a new frame before each child xlator returns. Suppose you have 3 sub-xlator, that is, 3 sub-volumes. The next is only the same as the first new frame.

According to the thread library model, call_stack_t corresponds to the thread control block.

Useful member variables are:

Uid and gid indicate who the calling frame comes from and are used for authentication and authentication.

Pid represents the process pid that initiated this request.

Trans points to the transport structure associated with the request. It is used on the server side to implement node-based access control, or to track requests along failed connections. What is it used for in client?

Frames: false frame, pointing to the first real frame of the request.

Op / type?

Call_pool_t?

STACK_WIND and STACK_UNWIND is equivalent to function calls and returns. In the execution of this xlator request, the allocation table is executed first, and then ing is stopped after wind. When the function rfn callback function executes, this operation begins to resume execution. Rfn may be called multiple times. (multiple subvolumes).

Todo

Call stack

Call stub

Version 1.0

Last updated 2014-04-30 12:28:09 CST

That's all for the content of "how to use GlusterFS Translator API". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.