Shulou(Shulou.com)06/02 Report--

This article mainly introduces "airflow uses mysql database and LocalExecutor concurrent scheduling". In daily operation, I believe many people have doubts about airflow using mysql database and LocalExecutor concurrent scheduling. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts of "airflow using mysql database and LocalExecutor concurrent scheduling". Next, please follow the editor to study!

Mysql-airflow executes on mysql

Create database airflow;-create a database

GRANT all privileges on airflow.* TO 'airflow'@'%' IDENTIFIED BY' 123456login;-- Grant all permissions of the database airflow to the airflow user with a password of 123456 and the user can log in to any IP segment

FLUSH PRIVILEGES;-refresh permissions

Set explicit_defaults_for_timestamp = 1

Modify airflow configuration file airflow.cfg

Sql_alchemy_conn = mysql://airflow:123456@mysqlIP/airflow

Initialize the database

Airflow initdb

If you have previously initialized with sqllite, you can use airflow resetdb

Airflow-LocalExecutor

Modify the executor to schedule tasks concurrently with airflow

Modify the configuration file airflow.cfg

Executor = LocalExecutor

Restart scheduler (default concurrent scheduling 16)

Distributed problem

1. Need to use synchronization method to solve the synchronization problem between dags directory and worker-- to be solved

two。 To solve the single point problem of scheduler, please refer to apache-airflow scheduling system https://blog.csdn.net/somezz/article/details/82749495

At this point, the study on "airflow uses mysql database and LocalExecutor concurrent scheduling" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.