Shulou(Shulou.com)06/01 Report--

Python cross-database check data consistency example analysis, in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Because the samza real-time streaming platform has recently been launched to transfer data from the source side to the data warehouse in real time, so you need to check the accuracy of samza data transfer, then check the number of rows of the two data-side tables regularly through the following python script to simply check the data consistency. The following script runs every day at 6 a.m., checks that all rows of data before yesterday are the same, and notifies you by email. Because there are many tables to be detected and the cleaning rules are inconsistent, we can only compare one table at a time. The following is only part of the script code.

# coding=utf-8

Import MySQLdb

Import psycopg2

Import smtplib

Import time

From email.mime.text import MIMEText

# General mail functions

Def mail (sub,content):

Mailto_list= ["hzwuj@tairanchina.com"]

Mail_host= "smtp.tairanchina.com"

Mail_uer= "trcloud@tairanchina.com"

Mail_pass= "ringing Heights 3OAGs5TGeh"

Message = MIMEText (content,_charset='utf-8')

Message ['Subject'] = sub

Message ['From'] = mail_uer

Message ['To'] = ";" .join (mailto_list)

Try:

S = smtplib.SMTP ()

S.connect (mail_host)

S.login (mail_uer,mail_pass)

S.sendmail (mail_uer, mailto_list, message.as_string ())

S.close ()

Return True

Except Exception:

Print 'filed'

Return False

# set the variable to today's date

Datetime=time.strftime ("% Y%m%d", time.localtime ())

# get the total number of cleaned entries from the source database mysql

Conn=MySQLdb.connect (host='115.231.97.10',port=3306,user='select',passwd='123456',db='cms')

Cursor1=conn.cursor (cursorclass=MySQLdb.cursors.DictCursor)

Cursor1.execute ("select 'biz_account' as tablename,count (*) as sum from biz_account where del_flag='0' and date (create_date))

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.