Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "how to manage multimedia files through Git". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The problem of managing multimedia files by Git

As we all know, Git is not very good for dealing with non-text files, but that doesn't stop us from trying. Here is an example of using Git to copy a photo file:

$du-hs108K. $cp ~ / photos/dandelion.tif. $git add dandelion.tif$ git commit-m'added a photo' [master (root-commit) fa6caa7] two photos 1 file changed, 0 insertions (+), 0 deletions (-) create mode 100644 dandelion.tif$ du-hs1.8M.

Nothing unusual so far. Add a 1.8MB photo to a directory so that the directory becomes the size of 1.8MB. So next, we try to delete the file.

$git rm dandelion.tif$ git commit-m'deleted a photo'$ du-hs828K.

Here we can see some problems: deleting a file that has already been committed will still increase the size of the repository to eight times the original size (from 108K to 828K). We can test it many times to get a better average, but this simple demonstration is consistent with my experience. Submitting non-text files takes less space at the beginning, but the longer a project is active, the more people may modify the static content, and more bits and pieces of files will be added together. When a Git repository gets bigger and bigger, the main cost is often speed. The time to pull and push, from the time you first took a sip of coffee to the time you think you may be off the Internet.

What is the reason for the growing size of the Git repository due to static content? Files that are made up of text allow Git to pull only those modified parts. Raster images and music files are different from text for Git files. You can take a look at the binary data in .png and .wav files. So, Git simply takes all the data and creates a new copy, even if a picture is modified by only one pixel.

Git-portal

In practice, many multimedia projects do not need or do not want to track the history of media. The media part of a project generally has a different life cycle than the text or code part. Media resources are generally generated in one direction: a picture starts with a pencil draft and reaches its destination in the form of a digital painting. Then, although the text can be rolled back to the earlier version, the work of art will only move forward. The media in the project is rarely bound to a specific version. The exception is usually a graph that reflects a dataset, usually a table, graph, or chart that can be done in a text-based format such as SVG.

So, in many projects that contain both text (whether narrative prose or code) and media, Git is an acceptable solution for document management, as long as there is a playground outside the version control cycle for artists to play.

Graphic showing relationship between art assets and Git

An easy way to enable this feature is Git-portal, a Bash script with Git hooks that moves static files out of the scope of Git from a folder and replaces them with symbolic links. Git submits linked files (sometimes called aliases or shortcuts), which are smaller symbolic link files, so all submissions are text files and links that represent media files. Because the stand-in files are symbolic links, the project will also run as expected, because the local machine will process them and convert them into "real" copies. When replacing files with symbolic links, Git-portal maintains the structure of the project, so if you think that Git-portal is not suitable for your project, or if you need to build a version of the project without symbolic links (for distribution, for example), you can easily reverse the process.

Git-portal also allows remote synchronization of static resources through rsync, so users can set a remote storage location as a central authorization source.

Git-portal is an ideal solution for multimedia engineering. Similar multimedia projects include video games, desktop games, virtual reality projects that require large 3D model rendering and texture, books with graphics and .odt output, collaborative blog sites, music projects, and so on. It is not uncommon for artists to perform versioning in applications in the form of layers (in the graphics world) and tracks (in the music world)-so Git does not add anything to the multimedia project file itself. The functions of Git can be used in other parts of the art project (such as prose and narration, project management, subtitle files, thanks, marketing copies, documents, etc.), while the structured remote backup function is used by artists.

Install Git-portal

Git-portal 's RPM installation package is located in https://klaatu.fedorapeople.org/git-portal and can be used for download and installation.

In addition, users can install it manually from the Gitlab home page of Git-portal. This is just a Bash script and some Git hooks (also Bash scripts), but it takes a quick build process to let it know where to install.

$git clone https://gitlab.com/slackermedia/git-portal.git git-portal.clone$ cd git-portal.clone$. / configure$ make$ sudo make install uses Git-portal

Git-portal is used with Git. This means that, as with all large file extensions in Git, there are some extra steps to remember. However, you only need to use Git-portal when dealing with your media resources, so it's easy to remember unless you treat large files as text files (rare for Git users). One of the installation steps that must be done to use Git-portal is:

$mkdir bigproject.git$ cd! $$git init$ git-portal init

Git-portal 's init function creates a _ portal folder in the Git repository and adds it to the .gitignore file.

On weekdays, the use of Git-portal and Git collaboration is very smooth. A good example is a MIDI-based music project: the project files generated by the music workstation are text-based, but the MIDI files are binary data:

$ls-1_portalsong.1.qtrsong.qtrsong-Track_1-1.midsong-Track_1-3.midsong-Track_2-1.Mid $git add song*qtr$ git-portal song-Track*mid$ git add song-Track*mid

If you look at the _ portal folder, you will find the original MIDI file there. These files are replaced in their original location with a link to _ portal, allowing the music workstation to run as expected.

$ls-lG [...] _ portal/ [...] Song.1.qtr [...] Song.qtr [...] Song-Track_1-1.mid-> _ portal/song-Track_1-1.Mid * [...] Song-Track_1-3.mid-> _ portal/song-Track_1-3.Mid * [...] Song-Track_2-1.mid-> _ portal/song-Track_2-1.Mid *

Like Git, you can also add files in a directory.

$cp-r ~ / synth-presets/yoshimi. $git-portal add yoshimiDirectories cannot go through the portal. Sending files instead.$ ls-lG _ portal/yoshimi [...] Yoshimi.stat->.. / _ portal/yoshimi/yoshimi.stat*

Deletion works as expected, but when removing something from _ portal, you should use git-portal rm instead of git rm. Use Git-portal to ensure that files are deleted from _ portal:

$ls_portal/ song.qtr song-Track_1-3.mid@ yoshimi/song.1.qtr song-Track_1-1.mid@ song-Track_2-1.middie $git-portal rm song-Track_1-3.midrm 'song-Track_1-3.middie $ls_portal/ song-Track_1-1.midi * song-Track_2-1.mid@ yoshimi/

If you forget to use Git-portal, you need to manually delete the file under _ portal:

$git-portal rm song-Track_1-1.midrm 'song-Track_1-1.middie $ls _ portal/song-Track_1-1.mid* song-Track_2-1.midi * yoshimi/$ trash _ portal/song-Track_1-1.mid

The only other function of Git-portal is to list all current links and find symbolic links that may have been damaged. Sometimes this occurs when the files in the project folder are moved:

$mkdir foo$ mv yoshimi foo$ git-portal statusbigproject.git/song-Track_2-1.mid: symbolic link to _ portal/song-Track_2-1.midbigproject.git/foo/yoshimi/yoshimi.stat: broken symbolic link to.. / _ portal/yoshimi/yoshimi.stat

If you use Git-portal for private projects and maintain your own backups, that's all you need to know about Git-portal. If you want to add a collaborator or you want Git-portal to manage backups like Git, you can create a remote location.

Add Git-portal remote location

Adding a remote location to Git-portal is achieved through the existing remote functionality of Git. Git-portal implements Git hooks (scripts hidden in the repository .git folder) to find out if folders starting with _ portal exist on your remote location. If it finds one, it tries to use rsync to synchronize files with the remote location. Git-portal performs this operation when the user is doing Git push and Git merge (or when Git pull is actually a fetch and automatic merge).

If you only cloned the Git repository, you may never add a remote location yourself. This is a standard Git process:

$git remote add origin git@gitdawg.com:seth/bigproject.git$ git remote-vorigin git@gitdawg.com:seth/bigproject.git (fetch) origin git@gitdawg.com:seth/bigproject.git (push)

The name origin is a popular convention for your main Git repository, and it makes sense to use it for Git data. However, your Git-portal data is stored separately, so you have to create a second remote location so that Git-portal knows where to push and where to pull. Depending on your Git host, you may need a separate server because Git hosts with limited space are less likely to accept GB-level media assets. Or, maybe your server only allows you access to your Git repository, not external storage folders:

$git remote add _ portal seth@example.com:/home/seth/git/bigproject_portal$ git remote-vorigin git@gitdawg.com:seth/bigproject.git (fetch) origin git@gitdawg.com:seth/bigproject.git (push) _ portal seth@example.com:/home/seth/git/bigproject_portal (fetch) _ portal seth@example.com:/home/seth/git/bigproject_portal (push)

You may not want to provide all users with personal accounts on the server, and you don't have to do so. To provide access to the server that hosts the repository's large file assets, you can run a Git front end, such as Gitolite, or you can use rrsync (restricted rsync).

Now you can push your Git data to your remote Git repository and send your Git-portal data to your remote portal:

$git push origin HEAD master destination detectedSyncing _ portal content...sending incremental file listsent 9305 bytes received 18 bytes 1695.09 bytes/sectotal size is 60358015 speedup is 6474.10Syncing _ portal content to example.com:/home/seth/git/bigproject_portal

If you have installed Git-portal and configured the remote location of _ portal, your _ portal folder will be synchronized, get new content from the server, and send new content with each push. Although you don't need to Git commit or push to synchronize with the server (users can use rsync directly), I find submission useful for artistic content changes. This will integrate artists and their digital assets into the rest of the workflow and provide useful metadata about the progress and speed of the project.

Other options

If Git-portal is too easy for you, there are other options for Git to manage large files. Git large File Storage (LFS) is an offshoot of a shutdown project called git-media, which is maintained and supported by GitHub. It requires special commands (such as git lfs track to protect large files from Git tracking) and requires users to maintain a .gitattributes file to update which files in the repository are tracked by LFS. For large files, it only supports HTTP and HTTPS remote hosts. So you have to configure the LFS server so that users can authenticate through HTTP instead of SSH or rsync.

Another more flexible option than LFS is git-annex. You can learn more about the big binary blob in my article Managing Git (ignore the chapter on git-media, an abandoned project, because its flexibility has not been continued by its successor Git LFS). Git-annex is a flexible and elegant solution. It has a delicate system for adding, deleting, and moving large files in the repository. Because it is flexible and powerful, there are many new commands and rules to learn, so it is recommended to take a look at its documentation.

However, if your requirements are simple and you may prefer solutions that integrate existing technologies for simple and obvious tasks, then Git-portal may be a more appropriate tool for work.

This is the end of "how to manage multimedia files through Git". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.