Shulou(Shulou.com)06/02 Report--

This article will explain in detail about Java one-time query hundreds of thousands or millions of data solutions, the quality of the article content is high, so Xiaobian share for everyone to make a reference, I hope you have a certain understanding of related knowledge after reading this article.

Java query one-time query hundreds of thousands, millions of data solutions.

One of the early ways to write tools was to package millions of data into rar files.

So it was a stupid idea. I hope that the master guidance, there is no good way ah.

First, check out all the data in batches. In the example, it is a batch of 10,000 items.

After finding out the data, the data of each time is stored in the local file according to certain rules.

When acquiring data, batch reading is used to obtain large quantities of data. See http://www.example.com for this method. yijianfengvip.blog.163.com/blog/static/175273432201191354043148/

The following is a query database. Query by batch

public static void getMonthDataList() { ResultSet rs = null; Statement stat = null; Connection conn = null; List list = new ArrayList(); try { conn = createConnection(); if(conn!= null){ SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd"); SimpleDateFormat timesdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); String nowDate = sdf.format(new Date()); Config.lasttimetext = timesdf.format(new Date()); String lastDate = sdf.format(CreateData.addDaysForDate(new Date(), 30)); stat = conn.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE,ResultSet.CONCUR_UPDATABLE); int lastrow = 0; int datanum = 0; String countsql = "SELECT count(a.id) FROM trip_special_flight a" + " where a.dpt_date >= to_date('"+nowDate+"','yyyy-mm-dd') " + "and a.dpt_date "+lastrow+" order by a.get_time desc"; rs = stat.executeQuery(countsql); while (rs.next()) { datanum = rs.getInt(1); } int onerun = 10000; int runnum = datanum%onerun==0? (datanum/onerun):(datanum/onerun)+1; for(int r =0;r10){ FileUtil.appendToFile(Config.tempdatafile, text); } lastrow+=onerun; } } } catch (Exception e) { e.printStackTrace(); } finally { closeAll(rs, stat, conn); } }

----java one-time query hundreds of thousands, millions of data solution

After storing temporary files, read a large number of data files.

Set cache size BUFFER_SIZE, Config.tempdatafile is the file address.

Source Blog http://yijianfengvip.blog.163.com/blog/static/175273432201191354043148/

package com.yjf.util; import java.io.File; import java.io.RandomAccessFile; import java.nio.MappedByteBuffer; import java.nio.channels.FileChannel; public class Test { public static void main(String[] args) throws Exception { final int BUFFER_SIZE = 0x300000; //buffer is 3M File f = new File(Config.tempdatafile); //Source blog yijianfengvip.blog.163.com/blog/static/175273432201191354043148/ int len = 0; Long start = System.currentTimeMillis(); for (int z = 8; z >0; z--) { MappedByteBuffer inputBuffer = new RandomAccessFile(f, "r") .getChannel().map(FileChannel.MapMode.READ_ONLY, f.length() * (z-1) / 8, f.length() * 1 / 8); byte[] dst = new byte[BUFFER_SIZE]; for (int offset = 0; offset

= BUFFER_SIZE) { for (int i = 0; i < BUFFER_SIZE; i++) dst[i] = inputBuffer.get(offset + i); } else { for (int i = 0; i < inputBuffer.capacity() - offset; i++) dst[i] = inputBuffer.get(offset + i); } int length = (inputBuffer.capacity() % BUFFER_SIZE == 0) ? BUFFER_SIZE : inputBuffer.capacity() % BUFFER_SIZE; len += new String(dst, 0, length).length(); System.out.println(new String(dst, 0, length).length()+"-"+(z-1)+"-"+(8-z+1)); } } System.out.println(len); long end = System.currentTimeMillis(); System.out.println("File cost: " + (end-start) + "ms"); } }

Method for reading large data files.

About Java one-time query hundreds of thousands or millions of data solutions to share here, I hope the above content can be of some help to everyone, you can learn more knowledge. If you think the article is good, you can share it so that more people can see it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.