How to create the connector of kafka automatically by combining python with shell
Most people do not understand the knowledge points of this article "python how to combine shell to automatically create a kafka connector", so the editor summarizes the following contents, detailed contents, clear steps, and has a certain reference value. I hope you can get something after reading this article. Let's take a look at this "how to combine python with shell to automatically create a kafka connector" article.
Environment cat / etc/redhat-release CentOS Linux release 7.5.1804 (Core) [root@localhost ~] # uname-aLinux localhost.localdomain 3.10.0-862.el7.x86_64 # 1 SMP Fri Apr 20 16:44:24 UTC 2018 x 86 "64 GNU/Linuxpython-VPython 2.7.5 install python package connected to oracle pip install cx_Oracle==7.3 get oracle table information cat query_oracle.py #! / usr/bin/env pythonimport cx_Oracleimport sysimport osimport csvimport tracebackfile = open (" oracle.txt ") Close () user = "test" passwd = "test" listener = '10.0.2.15:1521/orcl'conn = cx_Oracle.connect (user, passwd) Listener) cursor = conn.cursor () sql = "select table_name from user_tables" cursor.execute (sql) LIST1= [] while True: row = cursor.fetchone () if row = = None: break for table in row: # print table LIST1.append (table) LIST2= [] for i in LIST1: sql3 = "select COLUMN_NAME,DATA_TYPE,DATA_PRECISION DATA_SCALE from cols WHERE TABLE_name=upper ('% s') "I cursor.execute (sql3) cursor.execute (sql3) row3 = cursor.fetchall () for data in row3: # LIST2.append (I) LIST2.extend (list (data)) LIST2.append (I) f=open ('oracle.txt','a+') print > > fame LIST2 LIST2= [] # f=open (' test.txt','a+') # select table_name,column_name DATA_TYPE from cols WHERE TABLE_name=upper ('student') # select column_name,DATA_TYPE from cols WHERE TABLE_name=upper ('student') Remove the redundant cat auto.sh #! / bin/bash#python query_oracle.py | tr ","'| tr "'"'| tr "["| tr"] ""# > oracle.txt > oracle_tables.txtcat oracle.txt | tr" [] '"" | sed "s # [] [] * # # g" | sed's / ^ [\ t] * / / g' > > oracle_tables.txtcat oracle_tables.txt SNO NUMBER 19 0 SNAME VARCHAR2 None None SSEX VARCHAR2 None None SBIRTHDAY DATE None None SCLASS VARCHAR2 None None STUDENT DATE_DATE SNO2 NUMBER 19 0 SNAME VARCHAR2 None None SSEX VARCHAR2 None None SBIRTHDAY DATE None None SCLASS VARCHAR2 None None STUDENT2 INPUT_TIMESNO3 NUMBER 19 2 SNAME VARCHAR2 None None SSEX VARCHAR2 None None SBIRTHDAY DATE None None SCLASS VARCHAR2 None None STUDENT3 DATA_DATEshell foot This processing table information file cat connect.sh#! / bin/bash# gets the number of lines of temporary files FILE_NUM=$ (cat oracle_tables.txt | egrep-v'# | ^ $'| wc-l) # clear the script for automatically creating connectors > create-connect.sh# circular temporary files each line for i in `seq $FILE_ NUM`do FILE_LINE=$ (sed-n ${I} p oracle_tables.txt) TABLE_NAME=$ (echo ${FILE_LINE} | sed's / [\ t] * $/ / g' | awk'{print $(NF-1)}') COL_NUM=$ (echo ${FILE_LINE} | sed's / [\ t] * $/ / g' | awk-F "[]"'{print NF}') REAL_COL_NUM= `expr $COL_NUM-2` # clear temporary files > ${TABLE_NAME} .txt > ${TABLE_NAME} .sql # looping temporary files per line Column for j in `expr $j + 1`m = `expr $j + 2`n = `expr $j + 3` COL_NAME=$ (echo $FILE_LINE | cut-d ""-f ${j}) COL_DATA_TYPE=$ (echo $FILE_LINE | cut-d "- f ${k}) COL_DATA_PRECISION=$ (echo $FILE_LINE | cut-d"-f ${m}) COL_DATA_SCALE=$ (echo $FILE_LINE | cut-d ""-f ${n}) # determines whether the data type of the column is NUMBER if ["$COL_DATA_TYPE" = "NUMBER"] then # CAST (* AS *) AS * part of the circular stitching SQL query Append to the temporary file echo "CAST ($COL_NAME AS $COL_DATA_TYPE ($COL_DATA_PRECISION,$COL_DATA_SCALE)) AS $COL_NAME" > > ${TABLE_NAME} .txt else # column names in the circular stitching SQL query Append to temporary file echo "$COL_NAME" > > ${TABLE_NAME} .txt fi done # to assemble the complete SQL statement, append to temporary file echo "select $(cat ${TABLE_NAME} .txt | tr"\ n "," | sed-e's / ) from $TABLE_NAME where $(sed-n ${I} p oracle_tables.txt | cut-d''- f$COL_NUM) > = trunc (sysdate-2) and $(sed-n ${I} p oracle_tables.txt | cut-d'- f$COL_NUM) = trunc (sysdate-2) and $(sed-n ${I} p oracle_tables.txt | cut-d'- f$COL_NUM) = trunc (sysdate-2) and $(cat. / TABLE_TIME/) ${TABLE_NAME} _ TIME_COL.txt)
