Shulou(Shulou.com)06/01 Report--

Today, I would like to share with you how to check python text similarity calculation simhash source code related knowledge points, detailed content, clear logic, I believe that most people still know too much about this knowledge, so share this article for your reference, I hope you can get something after reading this article, let's take a look at it.

Scene:

1. Calculate the Simhash value and Hamming distance.

2.SimHash is suitable for the similarity comparison of longer texts (more than 300 to 500 words). The shorter the text is, the higher the misjudgment rate is.

Python implementation:

The code is as follows

#-*-encoding:utf-8-*-import mathimport jiebaimport jieba.analyseclass SimHash (object): def getBinStr (self, source): if source = = "": return 0 else: X = ord (source [0]) 0: tmp = 1 ret + = str (tmp) return ret def sim_hash (self Rawstr): seg = jieba.cut (rawstr) keywords = jieba.analyse.extract_tags ("|" .join (seg), topK=100, withWeight=True) ret = [] for keyword Weight in keywords: binstr = self.getBinStr (keyword) keylist = [] for c in binstr: weight = math.ceil (weight) if c = "1": keylist.append (int (weight)) else: keylist.append (- int (weight)) Ret.append (keylist) # dimensionality reduction rows = len (ret) cols = len (ret [0]) result = [] for i in range (cols): tmp = 0 for j in range (rows): tmp + = int (rett [j] [I]) if tmp > 0: Tmp = "1" elif tmp

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.