Shulou(Shulou.com)11/24 Report--

April 20 (Xinhua) over the past four months, artificial intelligence chatbots have become more and more popular, and their ability to perform a variety of tasks, such as writing complex academic papers and conducting intense conversations, is amazing.

Chatbots don't think like humans. They don't even know what they're talking about. They can imitate human language because the artificial intelligence that drives them has absorbed a large amount of text, most of which is crawled from the Internet.

These texts are the main sources of world information in the construction process of artificial intelligence, and they will have a far-reaching impact on the response of artificial intelligence. If artificial intelligence does well in judicial exams, it may be because its training data contain thousands of LSAT (Law School Admission Test) materials.

Technology companies always keep secret about what information they provide to artificial intelligence. As a result, the Washington Post began to analyze one of the important data sets, thoroughly revealing the proprietary, personal, and often offensive types of websites used to train AI.

To explore the internal composition of artificial intelligence training data, the Washington Post worked with researchers at the Allen Institute of artificial Intelligence to analyze Google's C4 data set. This data set is a massive snapshot of more than 15 million websites that are used to train many high-profile English artificial intelligence, such as Google's T5 and Facebook's LLaMA. OpenAI did not disclose what data sets they used to train models that support the chatbot ChatGPT.

In the survey, the researchers classified websites using data from Similarweb, a web analytics company. About 1/3 of these sites cannot be classified and are excluded, mainly because they no longer exist on the Internet. The researchers then ranked the remaining 10 million sites according to the number of "token" that appeared on each site in the data set. A token is a small piece of text that processes information, usually a word or phrase, used to train the AI model.

From Wikipedia to WoWhead

C4 dataset websites are mainly from news, entertainment, software development, medical and content creation and other industries. This may explain why these areas may be threatened by a new wave of artificial intelligence. The top three sites are: Google Patent search, which contains patent texts published around the world, Wikipedia, and Scribd, a digital library that only accepts paid subscriptions. In addition, other sites at the top of the list are the pirated e-book market Library (190), which was shut down by the U.S. Department of Justice for illegal acts. In addition, there are at least 27 websites in the data set that are identified by the US government as markets for pirated and counterfeit products.

Other top websites are also available, such as World of Warcraft player forum wowhead (No. 181) and thriveglobal, a website created by Arianne Huffington (Arianna Huffington) to help ease job burnout (No. 175). In addition, there are at least 10 websites that sell trash bins, including dumpsteroid (183rd), but it no longer seems to be accessible.

Although most websites are secure, some sites have serious privacy issues. For example, there are two websites that rank in the top 100, both of which privately host copies of state voter registration databases. Although voter data are public, these models may use this personal information in unknown ways.

Industrial and commercial websites account for the largest category (16% of the classified token). At the top of the list is The Motley Fool, which provides investment advice (No. 13). The second is Kickstarter, which allows users to crowdfund creative projects (No. 25). Lower-ranked Patreon, at 2398, helps creators charge monthly fees from subscribers to get exclusive content.

However, Kickstarter and Patreon may give artificial intelligence access to artists' ideas and marketing copywriting, and there are concerns that AI may copy these works when providing advice to users. At present, artists do not receive any compensation when their works are included in artificial intelligence training data, and they have filed infringement claims against text-to-image generators Stable Diffusion, MidJourney and DeviantArt.

According to the Washington Post's analysis, more legal challenges may be coming: there are more than 200 million copyright symbols in the C4 data set (meaning works registered as intellectual property rights).

Technology websites are the second largest category, accounting for 15 per cent of the classified token. This includes a number of platforms that help people build websites, such as the Google collaboration platform (No. 85), whose pages range from the Reading Judo Club in England to kindergartens in New Jersey.

The C4 dataset also contains more than 500,000 personal blogs, accounting for 3.8% of the classified content. Publishing platform Medium ranks 46th and is the fifth largest technology website, with tens of thousands of blogs under its domain name. In addition, there are blogs written on WordPress, Tumblr, Blogpot and Live Journal.

These blogs come in a variety of forms, from professional to personal, such as a blog called "Grumpy Rumblings", co-written by two anonymous scholars, one of whom recently wrote about how their partner's unemployment affected husband and wife's taxes. In addition, there are some top blogs in the C4 dataset that focus on live-action role-playing games.

The content of social networks such as Facebook and Twitter, which are seen as the core of modern networks, is banned, meaning that most of the data sets used to train artificial intelligence cannot access them. Tech giants such as Facebook and Google are sitting on huge amounts of conversational data, but they don't know how to use personal user information to train artificial intelligence models that are used internally or sold as products.

News and media sites rank third in all categories, and half of the top 10 sites are news media: the New York Times ranked fourth, the Los Angeles Times ranked sixth, the Guardian ranked seventh, Forbes ranked eighth, the Huffington Post ranked ninth and the Washington Post ranked 11th. Like artists and creators, a number of news organizations have criticized technology companies for using their content without authorization or compensation.

At the same time, the Washington Post found that several media outlets ranked low in NewsGuard's independent credibility rating: Russia's RT (65th), far-right news site breitbart (159th) and vdare, an anti-immigrant website related to white supremacism (993rd).

Chatbots have been shown to be able to share misinformation. Unreliable training data may cause them to spread biases and misinformation, while users cannot trace their original sources.

Community sites account for about 5% of the classified content.

What are the fish that the filter misses? Like most companies, Google filters and screens data before providing it to artificial intelligence. In addition to removing meaningless and repetitive words, the company also uses an open source "bad vocabulary list", including 402 English terms and an emoji. Companies often use high-quality data sets to fine-tune the model to shield users from what they don't want to see.

Although such lists are designed to limit models from being affected by racial defamation and undesirable content during training, many things are screened by filters. The Washington Post found hundreds of porn sites and more than 72000 "Nazi" examples, all on the list of banned words.

At the same time, the Washington Post found that the filters failed to remove some disturbing content, including white supremacist websites, anti-transgender sites and 4chan, an anonymous message board known for organizing activities aimed at personal harassment. Websites promoting conspiracy theories were also found in the study.

Is your website used to train AI? Web crawling may sound like copying the entire Internet, but in fact it is just collecting snapshots, that is, crawling content from a sample of a web page at a particular time. The C4 dataset, originally created by the non-profit organization CommonCrawl, crawled web content in April 2019 and is a popular resource for artificial intelligence model training. CommonCrawl said the group tried to prioritize the most important and reputable sites, but did not try to avoid licensed or copyrighted content.

The Washington Post believes that it is important to present the full content of the data in artificial intelligence models that are expected to manage many aspects of people's modern lives. However, many sites in this dataset contain highly offensive language, and even if the model tries to cover up these words during training, objectionable content may still exist.

Experts say that although C4 data sets are large, large language models may use larger data sets. For example, OpenAI released GPT-3 training data in 2020, which is 40 times the amount of network crawling data in C4. GPT-3 's training data includes all English Wikipedia, free collections of unpublished writers often used by large technology companies, and a compilation of linked texts highly rated by Reddit users.

Experts say many companies do not even record the contents of the training data (or even internal data) for fear of finding identifiable personal information, copyrighted materials and other data stolen without consent. As the company highlights the challenges of explaining how chatbots make decisions, this is an area where executives need to give transparent answers.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.