Shulou(Shulou.com)06/02 Report--

At present, there are many isolated data islands, and it is difficult to connect business software or obtain data in software, especially the data crawling difficulty of C S software is greater.

The most common way of system docking is interface mode, which can be successfully docked under good luck, but interface docking mode often takes a lot of time to coordinate various software vendors.

In addition to the software interface, whether there are other ways, Xiaobian summarized the common data acquisition technologies for your reference, mainly divided into the following categories:

CS software data acquisition technology.

C/S architecture software belongs to relatively old architecture, and there are relatively few products that can collect data from this software.

A common example is Bo, a small software robot, which collects data on the interface based on "what you see is what you get" without the cooperation of software manufacturers. The output is a structured database or excel sheet. If you only need business data, or manufacturers go bankrupt, database analysis is difficult, this tool can collect data, especially the details page data collection function is more distinctive.

It is worth mentioning that the use threshold of this product is very low, and business students without IT background can also use it, greatly expanding the use of the crowd.

Second, network data collection API. Get data from websites through web crawlers and public APIs provided by some website platforms, such as Twitter and Sina Weibo APIs. This allows unstructured and semi-structured data to be extracted from web pages.

The whole process of web page big data collection and processing includes four main modules: web crawler (Spider), data processing (Data Process), crawling URL queue (URL Queue) and data.

database mode

The two systems have their own databases, and it is more convenient between databases of the same type:

1) If the two databases are on the same server, as long as there is no problem with the user name setting, they can directly access each other. You need to bring the database name and the schema owner of the table after the from. select * from DATABASE1.dbo.table1

2) If the databases of the two systems are not on a server, it is recommended to use the form of linked servers to process, or use openset and opendatasource, which requires access to the database for peripheral server configuration.

Connections between different types of databases are cumbersome and require a lot of settings to take effect, which are not explained in detail here.

The open database approach requires coordination of open databases of various software vendors, which is very difficult; if a platform has to connect databases of many software vendors at the same time and obtain data in real time, it is also a huge challenge to the performance of the platform itself.

Everyone is welcome to discuss together.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.