Shulou(Shulou.com)05/31 Report--

This article mainly introduces the way Python operates ES and data synchronization with Mysql related knowledge, the content is detailed and easy to understand, the operation is simple and fast, with a certain reference value, I believe that after reading this Python operation of ES and Mysql data synchronization method article will have a harvest, let's take a look at it.

There are two ways for Python to operate Elasticsearch # officially provided: can Elasticsearch# pip install elasticsearch# GUI:pyhon do graphical interface programming? -Tkinter-pyqt# uses (query is the key) # pip3 install elasticsearch https://github.com/elastic/elasticsearch-pyfrom elasticsearch import Elasticsearchobj = Elasticsearch (['127.0.0.1laza9200'],) # create Index (Index) # body: for what? Mapping: {}, setting: {} # result = obj.indices.create (index='user',ignore=400) # print (result) # Delete Index # result = obj.indices.delete (index='user',ignore= [400,404]) # insert and query data (additions, deletions, queries and modifications of documents) Is the most important # insert data # POST news/politics/1# {'userid':' 1, 'username':' lqz','password':'123'} # data = {'userid':' 1, 'username':' lqz','password':'123'} # result = obj.create (index='news', doc_type='politics', id=1 Body=data) # print (result) # Update data 'ActionRequestValidationException will be reported without doc package [Validation Failed: 1: script or doc is missing'''# data = {' doc': {'userid':' 1", 'username':' lqz','password':'123ee','test':'test'}} # result = obj.update (index='news', doc_type='politics', body=data, id=1) # print (result) # Delete data # result = obj.delete (index='news' Doc_type='politics' Id=1) # query # find all documents # query = {'query': {' match_all': {} # find all documents named jack # query = {'query': {' match': {'desc':' cute'} # query = {'query': {' term': {'from':' sheng'}} query = {'query': {' term': {'name':' wife'}} the difference between # term and match # term is a phrase query Will not be able to segment term things # match will be multi-match things for word segmentation Then query # find all documents older than 11 # allDoc = obj.search (index='lqz', doc_type='doc', body=query) allDoc = obj.search (index='lqz', doc_type='doc', body=query) print (allDoc) import jsonprint (json.dumps (allDoc)) # print (allDoc ['hits'] [' hits'] [0] ['_ source']) # how to integrate into the django project: create an index, just create it in advance # insert data Query data and modify data # query = {'query': {' term': {'name':' wife'}} # allDoc = obj.search (index='lqz', doc_type='doc', body=query) # json format directly returns # saas: software as a service, not using other people's services, but writing services for others to use-"normal development # public opinion monitoring system: (crawler) # only monitor Weibo -" Ikea: Weibo Baidu Tieba, listed company # Public Security: negative,-"tracking which user sent--" found # qq group, WeChat group-"Public opinion monitoring (third party can't do it, public opinion monitoring from Tencent, third-party organizations cooperate with Tencent, Tencent provides interface, third-party company does) # platform is developed, others buy services--" buy Weibo keyword monitoring for one year.

ERP: corporate finance, supply chain

Some big company, Kingdee, Yuyou, developed software-"your company buys its own server -" the software runs on your server.

Saas model: the company buys services, 10 years of services-"account password -" you can log in and operate-"find a user if something goes wrong--" the server is with someone else

-what the government spends money on-- "do you dare to reveal it?"

-Future Cloud Computing-Computing 1 +. + 100-"buy a computing service and get the result directly

# second use # https://github.com/elastic/elasticsearch-dsl-py# pip3 install elasticsearch-dslfrom datetime import datetimefrom elasticsearch_dsl import Document, Date, Nested, Boolean,analyzer, InnerDoc, Completion, Keyword, Text,Integerfrom elasticsearch_dsl.connections import connectionsconnections.create_connection (hosts= ["localhost"]) class Article (Document): title = Text (analyzer='ik_max_word', search_analyzer= "ik_max_word") Fields= {'title': Keyword ()}) author = Text () class Index: name =' myindex' # index name def save (self, * * kwargs): return super (Article Self) .save (* * kwargs) if _ _ name__ ='_ _ main__': # Article.init () # create mapping # save data # article = Article () # article.title = "test data" # article.author = "egon" # article.save () # data is saved # query data # s=Article.search () # s = s.filter ('match' Title= "Test") # results = s.execute () # # analogy queryset object Each object in the list is called Response in # # es, which is treated as a list Put an object in the list # print (results) # delete data # s = Article.search () # s = s.filter ('match', title= "test). Delete () # modify data s = Article (). Search () s = s.filter (' match' Title= "Test") results = s.execute () print (results [0]) results [0] .title = "xxx" results [0] .save () # other actions See documents mysql and Elasticsearch synchronize data # whenever a piece of data is inserted into the article table, it is automatically synchronized into the es # first option:-whenever a piece of data is inserted into the aritcle table (view class, Article.objects.create (), update)-insert one into the es-defect: code coupling is high, change a lot of places # the second option:-overwrite the create method Rewrite the update method-defects: synchronous operations-"inserts in es must return results in order to continue. # third option:-use celery, do async-defects: introduce celery, and message queues. # fourth option: (most used)-rewrite create method, rewrite update method, save with signal, asynchronous operation-defect: code intrusion # fifth scheme: (project does not write code, automatic synchronization), third-party open source plug-in-https://github.com/siddontang/go-mysql-elasticsearch----go write-you can rewrite one with python Put it on git for others to use (read mysql log)-independent of platform, independent of language-how to use:-source code download-- "cross-compile--" executable-- "run--" configuration file That's it # configuration file [[source]] schema = "database name" tables = ["article"] [[rule]] schema = "database name" table = "indicates" index = "index name" type = "type name" # defect:-es does not want to synchronize all fields of the table when synchronizing with mysql Mysql's multiple tables are directed to a type of es:-synchronize at first-use open source plug-ins (read mysql logs, connect to es, synchronize)-write by yourself with signals-and then at the high end: imitate his logic, written by python himself,-"(open source this thing) the use of haystack

A third-party module on django-"which django third-party modules have you used?"

Full-text search can be implemented on django

Equivalent to orm-- "docking es,solr,whoosh"

Https://www.yisu.com/article/218631.htm

Versions above es,6 are not supported

Full-text Retrieval based on haystack+Elasticsearch

Native operation of es: ELasticsearch Elasticsearch-dsl

Redis added that # 1 has only five data structures:-multiple data structures: string, hash, list, collection, ordered collection # 2 single thread, why is it so fast? -essentially because of the in-memory database-epoll model (io multiplexing)-single thread, no thread Communication between processes # 3 install redis# on linux download https://redis.io/download/ # decompress tar-xzf redis-5.0.7.tar.gz # establish a soft connection ln-s redis-5.0.7 redis cd redis make&&make install # bin path several commands: redis-cli,redis-server,redis-sentinel # can execute redis-server anywhere how to do? There are three ways to configure environment variable # 4 to start redis-mode 1: (generally not used, no configuration file)-redis-server-mode 2: (rarely used) redis-serve-- port 6380-mode 3: (all use this, configuration file) whether daemonize yes # starts the pidfile / var/run/redis.pid # process number with the daemon Delete port 637 port number dir "/ opt/soft/redis/data" # working directory logfile 6379.log # log location # start: redis-server redis.conf1#5 client connection redis-cli-h 127.0.0.1-p 6379log 6 usage scenario-look at the md documentation on "how Python operates ES and how to synchronize with Mysql" this article ends here Thank you for reading! I believe you all have a certain understanding of "how Python operates ES and how to synchronize data with Mysql". If you want to learn more, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.