Shulou(Shulou.com)06/03 Report--

Flink1.8 's support for hive is not good enough, resulting in 300W of data, unexpectedly read for 2 hours, planning to migrate the program to spark. Put the code on first. Later, it was found that sql should not have where conditions, and the speed was OK after removal.

Maven

Org.apache.hive hive-jdbc 1.1.0 org.apache.hadoop hadoop-common 3.1.2 jdk.tools jdk.tools 1.8 system ${JAVA_HOME} / lib/tools.jar

Java

Private final static String driverName = "org.apache.hive.jdbc.HiveDriver"; / / jdbc driver path private final static String url = ";"; / hive library address + library name private final static String user = ""; / / username private final static String password = "!"; / / password private final static String table= ""; private final static String sql = ""; public static void main (String [] arg) throws Exception {long time=System.currentTimeMillis () HttpClientUtil.sendDingMessage ("start synchronization hive-" + table+ ";" + Utils.getTimeString ()); / * * initialize environment * / final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment (); env.setParallelism (4); try {TypeInformation [] types = new TypeInformation [] {BasicTypeInfo.STRING_TYPE_INFO, BasicTypeInfo.STRING_TYPE_INFO} String [] colName = new String [] {"user", "name"}; RowTypeInfo rowTypeInfo = new RowTypeInfo (types, colName); JDBCInputFormatBuilder builder = JDBCInputFormat.buildJDBCInputFormat (). SetDrivername (driverName) .setDBUrl (url) .setUsername (user) .setPassword (password); Calendar calendar = Calendar.getInstance (); calendar.setTime (new Date ()) Calendar.add (Calendar.DATE,-1); / using yesterday's data SimpleDateFormat sj = new SimpleDateFormat ("yyyyMMdd"); String d=sj.format (calendar.getTime ()); JDBCInputFormat jdbcInputFormat = builder.setQuery (sql+ "and dt='" + d + "'limit 100000000") .setRowTypeInfo (rowTypeInfo). Finish (); DataSource rowlist = env.createInput (jdbcInputFormat) DataSet temp= rowlist.filter (new FilterFunction () {@ Override public boolean filter (Row row) throws Exception {String key=row.getField (0) .toString (); String value=row.getField (1) .toString (); if (key.length ())

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.