Shulou(Shulou.com)11/24 Report--

CTOnews.com The training of OpenAI's GPT model requires a large amount of network data, which may involve issues such as data privacy and copyright. To address these issues, OpenAI recently introduced a new feature that allows websites to prevent their web crawlers from scraping data from their websites to train GPT models.

CTOnews.com understands that web crawlers are automated programs that search and retrieve information on the Internet. OpenAI's web crawler, called GPTBot, visits various websites with a certain frequency and saves the content of the web pages for training GPT models.

OpenAI said in its blog post that website operators can prevent GPTBot from scraping data from their websites by blocking access to GPTBot in their Robots.txt file or by blocking their IP addresses. OpenAI also states that "web pages crawled using GPTBot user agents may be used to improve future models and filter out sources that require paid access, are known to collect personally identifiable information (PII), or have text that violates our policies." "For sources that don't meet the exclusion criteria," allowing GPTBot access to your website can help AI models become more accurate and improve their versatility and security. "

However, this does not retroactively remove content previously scraped from websites from ChatGPT's training data.

The Internet provides most of the training data for large language models, such as OpenAI's GPT model and Google's Bard, and acquiring data for AI training has become increasingly controversial. Some websites, including Reddit and Twitter, have taken steps to crack down on AI companies using their user posts for free, while some authors and other creators have filed lawsuits for alleged unauthorized use of their work.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.