Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is the principle of TF-IDF algorithm". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "what is the principle of TF-IDF algorithm"?

Concept

TF-IDF (term frequency-inverse document frequency)

It is a commonly used weighting technique for information retrieval and information mining.

TF-IDF is a statistical method to assess the importance of a word to one of the documents in a set of documents or a corpus.

The importance of a word increases in proportion to the number of times it appears in the document, but decreases inversely with the frequency of its appearance in the corpus.

Various forms of TF-IDF weighting are often used by search engines as a measure or rating of the correlation between files and user queries. In addition to TF-IDF, search engines on the Internet also use rating methods based on link analysis to determine the order in which documents appear in search results.

Principle

Word frequency (term frequency, TF)

Refers to the number of times a given word appears in a given document. This number is usually normalized (the numerator is generally smaller than the denominator and distinguished from IDF) to prevent it from biasing long files. (the same word may have a higher word frequency in a long file than a short one, regardless of whether the word is important or not. )

Inverse file frequency (inverse document frequency, IDF) is a measure of the universal importance of a word. The IDF of a particular word can be obtained by dividing the total number of files by the number of files containing the word, and then taking the logarithm of the resulting quotient.

The high frequency of words in a particular file, and the low frequency of the word in the entire file collection, can produce a high weight TF-IDF. Therefore, TF-IDF tends to filter out common words and retain important words.

Inverse file frequency (inverse document frequency,IDF)

A measure of the universal importance of a word. The IDF of a particular word can be divided by the total number of files divided by the number of files containing the word, and the resulting quotient is logarithmically obtained:

Where: | D |: total number of files in the corpus

Include words

The number of files (that is, if the word is not in the corpus, it will cause the divisor to be zero, so it is generally used

And then

The high frequency of words in a particular file, and the low frequency of the word in the entire file collection, can produce a high weight TF-IDF. Therefore, TF-IDF tends to filter out common words and retain important words.

The main idea of TFIDF is

If a word or phrase appears in an article with a high frequency of TF and seldom appears in other articles, it is considered that the word or phrase has a good ability to distinguish categories and is suitable for classification. TFIDF is actually: TF * IDF,TF word frequency (Term Frequency), IDF anti-document frequency (Inverse Document Frequency). TF indicates how often an entry appears in document d (another word: TF Term Frequency refers to the number of times a given word appears in the file). The main idea of IDF is: if there are fewer documents containing the entry t, that is, the smaller the n is and the larger the IDF is, it means that the entry t has a good ability to distinguish categories. If a certain type of document C contains the entry t, the number of documents containing t is m, while the total number of documents containing t in other categories is k, it is obvious that all the documents containing t are n=m+k, when m is big, n is also large, and the value of IDF obtained according to the IDF formula will be small, which means that the entry t category discrimination ability is not strong. (another said: IDF anti-document frequency (Inverse Document Frequency) means that the fewer documents that contain entries, the larger the IDF, which means that terms have a good ability to distinguish categories. In fact, however, if an entry appears frequently in a class of documents, it means that the entry can well represent the characteristics of the text of this class, and such entries should give them a higher weight. and selected as the feature word of this kind of text to distinguish it from other types of documents. This is the deficiency of IDF.

In a given document, termfrequency,TF refers to the frequency at which a given word appears in the file. This number is a normalization of the number of words (termcount) to prevent it from biasing long files. (the same word may have a higher number of words in a long file than a short one, regardless of whether the word is important or not. ) for words in a particular document, its importance can be expressed as:

The above formula is the number of times the word appears in the file, while the denominator is the sum of the times that all words appear in the file.

At this point, I believe you have a deeper understanding of "what is the principle of the TF-IDF algorithm?" you might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.