Shulou(Shulou.com)06/03 Report--

This article focuses on "how to use Flask to build ES search engine", interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to use Flask to build ES search engine.

1 Profil

Config.py

# coding:utf-8 import os DB_USERNAME = 'root' DB_PASSWORD = None # if there is no password, DB_HOST =' 127.0.0.1 'DB_PORT =' 3306 'DB_NAME =' flask_es' class Config: SECRET_KEY = "Random character" # Random SECRET_KEY SQLALCHEMY_COMMIT_ON_TEARDOWN = True # Auto commit SQLALCHEMY_TRACK_MODIFICATIONS = True # Auto sql DEBUG = True # Debug mode SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://%s:%s@%s:%s/%s'% (DB_USERNAME DB_PASSWORD,DB_HOST, DB_PORT, DB_NAME) # Database URL MAIL_SERVER = 'smtp.qq.com' MAIL_POST = 465 MAIL_USERNAME =' 3417947630 email qq.com 'MAIL_PASSWORD =' mailbox authorization code 'FLASK_MAIL_SUBJECT_PREFIX='M_KEPLER' FLASK_MAIL_SENDER=MAIL_USERNAME # default sender # MAIL_USE_SSL = True MAIL_USE_TLS = False

This is a relatively simple Flask Config file, of course, for the current project database connection is not necessary, I just use Mysql as an auxiliary, friends do not need to configure to connect to the database, there is enough ES. Then the mailbox notifies this to look at personal needs.

2 Log

Logger.py

Log module is an indispensable part of engineering application, so it is very necessary to output log files according to different production environments. In the words of the world, "if you don't have a log file, you don't know how to die."

# coding=utf-8 import os import logging import logging.config as log_conf import datetime import coloredlogs coloredlogs.DEFAULT_FIELD_STYLES = {'asctime': {' color': 'green'},' hostname': {'color':' magenta'}, 'levelname': {' color': 'magenta',' bold': False} 'name': {' color': 'green'}} log_dir = os.path.dirname (os.path.dirname (_ _ file__)) +' / logs' if not os.path.exists (log_dir): os.mkdir (log_dir) today = datetime.datetime.now (). Strftime ("% Y-%m-%d") log_path = os.path.join (log_dir, today + ".log") log_config = {'version': 1.0 # format output 'formatters': {' colored_console': {'format': "% (asctime) s -% (name) s -% (levelname) s -% (message) s",' datefmt':'% HRV% MRO% S'} 'detail': {' format':'% (asctime) s -% (name) s -% (levelname) s -% (message) slots, 'datefmt': "% Y-%m-%d% H:%M:%S" # time format},} 'handlers': {' console': {'class':' logging.StreamHandler', 'level':' DEBUG', 'formatter':' colored_console'}, 'file': {' class': 'logging.handlers.RotatingFileHandler',' maxBytes': 1024 * 1024 * 1024 'backupCount': 1,' filename': log_path, 'level':' INFO', 'formatter':' detail', # 'encoding':' utf-8', # utf8 Encoding prevents coding errors},} 'loggers': {' logger': {'handlers': [' console'], 'level':' DEBUG',},}} log_conf.dictConfig (log_config) log_v = logging.getLogger ('log') coloredlogs.install (level='DEBUG', logger=log_v) # # Some examples. # logger.debug ("this is a debugging message") # logger.info ("this is an informational message") # logger.warning ("this is a warning message") # logger.error ("this is an error message")

Here I have prepared a commonly used log configuration file, which can be used as a commonly used log format, can be directly called, according to different levels to output to the terminal or .log file, take away.

3 routing

For Flask projects, blueprints and routing make the entire project more enjoyable (code reading, of course).

Here, I use two branches as data support, one is the Math entry, and the other is the Baike entry. The data source is based on the Baidu encyclopedia crawler in the previous article, which is crawled according to the depth-first crawling method and put into the ES.

# coding:utf8 from flask import Flask from flask_sqlalchemy import SQLAlchemy from app.config.config import Config from flask_mail import Mail from flask_wtf.csrf import CSRFProtect app = Flask (_ _ name__,template_folder='templates',static_folder='static') app.config.from_object (Config) db = SQLAlchemy (app) db.init_app (app) csrf = CSRFProtect (app) mail = Mail (app) # do not import the registration blueprint before generating db. From app.home.baike import baike as baike_blueprint from app.home.math import math as math_blueprint from app.home.home import home as home_blueprint app.register_blueprint (home_blueprint) app.register_blueprint (math_blueprint,url_prefix= "/ math") app.register_blueprint (baike_blueprint,url_prefix= "/ baike") #-*-coding:utf-8-*-from flask import Blueprint baike = Blueprint ("baike" _ _ name__) from app.home.baike import views#-*-coding:utf-8-*-from flask import Blueprint math = Blueprint ("math", _ _ name__) from app.home.math import views

Declare a route and initialize it in the _ _ init__ file

Let's take a look at the implementation of routing (take Baike as an example)

#-*-coding:utf-8-*-import os from flask_paginate import Pagination, get_page_parameter from app.Logger.logger import log_v from app.elasticsearchClass import elasticSearch from app.home.forms import SearchForm from app.home.baike import baike from flask import request, jsonify, render_template, redirect baike_es = elasticSearch (index_type= "baike_data", index_name= "baike") @ baike.route ("/") def index (): searchForm = SearchForm () return render_template ('baike/index.html' SearchForm=searchForm) @ baike.route ("/ search", methods= ['GET',' POST']) def baikeSearch (): search_key = request.args.get ("b", default=None) if search_key: searchForm= SearchForm () log_v.error ("[+] Search Keyword:" + search_key) match_data = baike_es.search (search_key Count=30) # Flip PER_PAGE = 10 page= request.args.get (get_page_parameter (), type=int, default=1) start= (page-1) * PER_PAGE end= start + PER_PAGE total = 30 print ("most total big data:", total) pagination = Pagination (page=page, start=start, end=end) Total=total) context = {'match_data': match_data ["hits"] ["hits"] [start:end],' pagination': pagination, 'uid_link': "/ baike/"} return render_template (' data.html', q=search_key, searchForm=searchForm * * context) return redirect ('home.index') @ baike.route (' /') def baikeSd (uid): base_path = os.path.abspath ('app/templates/s_d/') old_file = os.listdir (base_path) [0] old_path = os.path.join (base_path Old_file) file_path = os.path.abspath ('app/templates/s_d/ {} .html' .format (uid)) if not os.path.exists (file_path): log_v.debug ("[-] File does not exist, renaming!!") Os.rename (old_path, file_path) match_data= baike_es.id_get_doc (uid=uid) return render_template ('s_d/ {} .html' .format (uid), match_data=match_data)

You can see that we successfully initialized the elasticSearch class and did a data search.

We use Flask's paging plug-in for paging and limit the number of pages per page, and jump to the details page according to Uid.

A careful little friend will find that I use a little trick here.

Baike.route ('/') def baikeSd (uid): base_path = os.path.abspath ('app/templates/s_d/') old_file = os.listdir (base_path) [0] old_path = os.path.join (base_path) Old_file) file_path = os.path.abspath ('app/templates/s_d/ {} .html' .format (uid)) if not os.path.exists (file_path): log_v.debug ("[-] File does not exist, renaming!!") Os.rename (old_path, file_path) match_data= baike_es.id_get_doc (uid=uid) return render_template ('s_d/ {} .html' .format (uid), match_data=match_data)

This ensures that only one html file is always left in the template that holds the details page.

4. Project launch

As always, it is convenient to use flask_script as the start-up scheme of the project.

# coding:utf8 from app import app from flask_script import Manager, Server manage = Manager (app) # launch command manage.add_command ("runserver", Server (use_debugger=True)) if _ _ name__ = "_ _ main__": manage.run ()

Type in black window

Python manage.py runserver

You can start the project, default port 5000, and access http://127.0.0.1:5000

Start using gunicorn

Gunicorn-c gconfig.py manage:app#encoding:utf-8 import multiprocessing from gevent import monkey monkey.patch_all () # number of parallel working processes workers = multiprocessing.cpu_count () * 2 + 1 debug = True reload = True # automatic reload loglevel = 'debug' # specify the number of threads per worker threads = 2 # forward to listening port 8000 bind =' 0.0.0.0virtual 5001' # set daemon Leave the process to supervisor management daemon = 'false' # working mode collaboration worker_class =' gevent' # set maximum concurrency worker_connections = 2000 # set process file directory pidfile = 'log/gunicorn.pid' logfile =' log/debug.log' # set access log and error message log path accesslog = 'log/gunicorn_acess.log' errorlog =' log/gunicorn_error.log'

Project screenshot

At this point, I believe you have a deeper understanding of "how to use Flask to build ES search engine". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.