Shulou(Shulou.com)11/24 Report--

Recently, an "unpopular singer" used an AI stand-in to sing songs in the Chinese music scene.

Overnight, "AI Stefanie Sun" became popular all over the Internet.

B station, AI Stefanie Sun cover Lin Junjie "she said", Jay Chou "Love in the Western Yuan Dynasty", Zhao Lei "Chengdu" and so on, so that a lot of netizens can not extricate themselves.

"unpopular singer" Stefanie Sun has become a hot singer of the year in 2023, setting off a lot of celebrity carnival.

Netizens said, "after listening to AI Stefanie Sun all night, I can't get out."

These cover songs are self-made and uploaded by UP hosts such as Eternity (L) and Rooster _ x through open source projects.

(the author seems to have deliberately added an one-second blank to the Peninsula Iron Box to make up 5 minutes and 20 seconds.)

UP owner: Eternity (besides AI Stefanie Sun, there are also AI Jay Chou, AI Wang Xinling, AI Lin Zhixuan.

Perhaps many people never dreamed that the Chinese music world would revive in this form in 2023.

Some time before the online business of "AI Stefanie Sun", a TikTok netizen wrote a song "Heart on My Sleeve" using AI and quickly went viral on the Internet, attracting more than 10 million spectators.

After listening to this song, netizens said one after another, "it surprised me, it was crazy!"

The song is written using the voices of two American pop musicians Drake and The Weeknd. First train AI through the singer's voice, and then use AI to create.

In China, bilibili's AI cover of Chinese music songs has gradually become the focus of attention of many people, Sun Yanzi, Wang Xinling, Jay Chou and other stars have "made a comeback."

And nothing is more popular than Stefanie Sun, who has directly become the new darling of AI by virtue of the title of "diva timbre".

UP master: Rooster _ x someone also made AI Stefanie Sun's Cantonese version of "Love comes too late".

However, for AI music production, it is not a new thing in the whole music industry. It's just that the fire of generative AI makes the threshold of AI cover be pulled down again.

Earlier this year, for example, Google launched a text-to-music model called MusicLM, which generates high-fidelity music at a frequency of 24 kHz by treating the music generation process as a hierarchical sequence-to-sequence modeling task.

For many fans, the AI cover satisfies many of their fantasies to some extent.

There are also some fans who have trained the AI of the late classic singers, including Assang, Zhang Guorong, Yao Bena, Teresa Teng and so on.

This may be a kind of digital immortality, in such a way to bring the long-lost voice back to people's hearts.

Midjourney's vivid ability to produce pictures has made people cry that the painter is going to lose his job. For AI cover, is the singer going to be replaced?

After a UP master @ A Zhang Rayzhang sang Killer Queen with his own timbre trained AI, he felt terrified instantly.

After making an emergency recording of a video, and attached "will AI singers make the cover section collectively unemployed?" I was killed by the AI version of me! "title.

Some netizens said that they were the first batch of AI victim painter and felt that they could not escape any career.

Some people also say that the cover is not like it at all in some places.

You know, for AI cover, it also needs rich timbre training data for specific artists, so that the works generated by AI are more real.

With regard to the current technology, although the singer's singing, skills and style can not be completely imitated, but the timbre can basically be completely reproduced.

But the real people can't be replaced.

AI cover is very popular, but the other side of music created by AI is a pressing copyright issue.

After AI's "Heart on My Sleeve" became popular on TikTok, the full version was uploaded to Apple Music, Spotify, YouTube and other platforms.

In this regard, American singer Drake expressed dissatisfaction on Ins, "this is the last straw." At present, the song has been removed from the shelves because of infringement.

Universal Music Group, which owns rights to superstars such as Taylor Swift and Bob Dylan, is urging Spotify and Apple to stop AI tools from grabbing lyrics and melodies from copyrighted songs by their artists, according to the Financial Times.

But some artists are not stingy with their voices, Musk's ex-girlfriend Grimes said online.

"anyone can use my voice AI to generate songs. "but you still have to pay another 50% of the copyright.

The author of the original project "so-vits-svc" behind the AI cover of the fire is also said to have deleted the project because too many people abused it.

SoVitsSvc: singing sound conversion

Project address: https://github.com/ svc-develop-team / so-vits-svc

The song-to-voice conversion model uses a SoftVC content encoder to extract the speech features of the source audio, and then feeds the vector directly into the VITS instead of converting it to a text-based intermediate format. As a result, both pitch and tone can be preserved.

In addition, the project developer also solved the problem of sound interruption by using NSF HiFiGAN as the vocoder (vocoder).

Change the feature input to Content Vec sampling rate and uniformly use 44100Hz

Due to the change of parameters and the simplification of the model structure, the GPU memory needed for reasoning is obviously reduced.

Added the option of automatic pitch prediction in 1:vc mode, which means that there is no need to manually enter pitch keys when switching voice, and the pitch of male and female voices can be changed automatically. However, this mode will cause pitch shift when changing songs.

Added option 2: reduce timbre leakage through k-means clustering scheme to make timbre more similar to target timbre.

Add option 3: add NSF-HIFIGAN enhancer, which has a certain sound quality enhancement effect on some models with few training sets, but has a negative effect on the trained models, so it is turned off by default.

The pre-training model file places the checkpoint_best_legacy_500.pt in the hubert directory.

Put G_0.pth and D_0.pth in the logs / 44k directory.

Pretreatment 0. Audio slicing

Using the audio-slicer-GUI or audio-slicer-CLI tool, slice the original audio to 5-15 seconds.

It's OK to be a little longer, but too long (for example, 30 seconds) can lead to "torch.cuda.OutOfMemoryError" during training or even preprocessing, commonly known as burst video memory.

After slicing, delete the audio that is too long and too short.

1. Resample to 44100Hz and mono

Python resample.py2. Automatically divide the data set into training set and verification set, and generate configuration files

Python preprocess_flist_config.py3. Generate hubert and f0

After python preprocess_hubert_f0.py completes the above steps, the dataset directory will contain the preprocessed data, and the dataset_raw folder can be deleted.

Now, you can modify some parameters in the generated config.json--

Keep_ckpts: keep the final keep_ckpts model during training. A setting of 0 retains all models, and the default is 3.

All_in_mem: loads all datasets into RAM. It can be enabled when the disk IO of some platforms is too low and the system memory is much larger than your dataset.

Train the python train.py-c configs/config.json-m 44k reasoning model to use "inference_main.py" when needed.

For example:

Python inference_main.py-m "logs/44k/G_30400.pth"-c "configs/config.json"-s "nen"-n "you know what you want to buy"-src.wav "- t 0 although the original project team has stopped maintenance, many netizens have carried out fork and made some updates.

For example, the following graphical interface:

Project address: https://github.com/ voicepaw / so-vits-svc-forkAI "Resurrection" AI cover, many netizens have done similar projects before, such as "AI-Talk" let Musk and Jobs have a time-travel conversation.

In the video, AI not only simulates their voices, but also simulates their dialogue ideas to some extent, making the communication process very smooth.

AI makes it possible for us to talk to the dead. Prior to this, bilibili UP mainly used AI to resurrect the granny.

For grandma's voice production, upload the existing audio directly, and the material basically comes from past phone recordings, videos or Wechat voice.

And use the audio editing software AU to adjust, the adjustment direction is mainly in noise reduction, human voice enhancement and so on.

Then the clearer audio samples are cut into short sentences of several seconds to facilitate tagging. Finally, the processed audio is packaged and put into the speech synthesis system.

Using the speech synthesis system, you can try to input text to speech.

Netizens have witnessed the song of Stefanie Sun, AI of science and technology, which has reached the hearts of many netizens.

Recently indulge in AI "cover", from AI Kanye singing punishing wine, down to Su Xiaoxing singing the truth is true. But to be serious, it is still the best cover of Stefanie Sun in AI.

Stefanie Sun, who is addicted to bilibili these days, has just listened to the song "one Game, one Dream". It is so beautiful that she sings it to her heart.

After listening to AI's cover songs, many netizens feel the horror of AI singers:

The power of science and technology is really frightening to think about.

I deeply feel the power of science and technology.

This is AI life, the number is skyrocketing!

There are also netizens' nostalgia for the deceased singer.

Reference:

Https://github.com/svc-develop-team/so-vits-svc

Https://www.bilibili.com/video/BV1io4y1w73k/?vd_source=eecf800392d116d832e90ad1c9ae70f6

This article comes from the official account of Wechat: Xin Zhiyuan (ID:AI_era)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.