Shulou(Shulou.com)06/02 Report--

In the movie Infernal Affairs, Tony Leung Chiu-wai plays an undercover policeman in the gang, although he has outstanding ability, but he also yearns for a peaceful life like ordinary people. When Officer Huang, who contacted him on a single line, was killed, his file at the police station was completely deleted by Liu Jianming, a fake policeman who was undercover in the police station. He was no longer able to prove his identity as a real policeman and was finally shot dead by a fake police officer.

A person who can never prove his identity has an empty body and cannot control the trajectory of his life. There is no doubt that undercover is brilliant or he will not be chosen, but in the end he will not be able to escape the control of others. Of course, this profession has also been marked with a lot of bright labels, # excellent ability # # Smart # # Super vitality # …...

In fact, I don't want to talk about undercover today, but I just want to bring out the crawler we want to talk about, because it can be compared to the "professional undercover" hidden in the online rivers and lakes: the super ability to obtain information, fate can not help, evil for good is entirely under the control of others, wavering. The only undeniable thing is that the industry takes a fancy to his "talent".

Burning its glory, reptiles are excellent.

Just like Chen Yongren, the undercover gangster in Infernal Affairs, he is alive because of his ability and excellence. The same is true of reptiles, the wind and water mixed in the world of online rivers and lakes:

1. Search engine crawler

Search engine crawlers are the most common well-intentioned crawlers. When users search for information with search engines, crawlers will automatically crawl relevant information from all kinds of websites instead of manually, and organize the index to present the really valuable content in front of users. As the higher the ranking of the website in the search engine, it will bring more visitors, and the traffic can convert value, so the major websites will scramble to welcome search engine crawlers to get more exposure. This kind of crawler serves not only users but also website service providers.

two。 Massive information processing

In the digital age, consulting companies can use crawler technology to quickly obtain large amounts of information, and can professionally process and analyze and classify massive information, so as to provide favorable data support for consulting services. use more data information samples to support more accurate decisions. Not only that, many enterprises will also use crawlers to monitor public opinion, which is of great help to find negative public opinion in time and do a good job in public relations crisis.

3. Substitute for repetitive labor force

In the network work, you will encounter a lot of repetitive work, such as deploying similar programs on hundreds of target machines. Excellent engineers always take the spirit of hack in their hearts, replace manpower with machines, liberate their hands, and write some automated scripts to support daily network deployment, testing, operation and maintenance, and so on. Programs that can automate a series of Web requests in batches and eventually achieve a certain purpose belong to "crawlers" in a broad sense. Such crawlers represent the idea that technology liberates people and gives people time to do more valuable things.

Reptiles are more willing to be kind to others, while showing their skills for justice, reptiles also have their own personality. The undercover in every story has always been "highly utilized" because their wisdom continues to create surprises and value for the party they work for, which makes it easier for him to be trusted. The whole thing about crawlers is to turn data into business opportunities:

1. Creativity standing on the shoulders of the industry

People always complain that it is difficult to start a business. Now there are only a few left after the liquidation of hundreds of group-buying companies, one of which is dominated by online taxi-hailing platforms that buy each other, and the P2P mutual fund industry has closed down due to policy regulation.

But there are also people who push through the old and bring forth the new, standing on the shoulders of the industry and playing with reptiles.

One play: the flow of "comparison of goods":

In the face of the dazzling products on the Internet, users will have difficulty in choosing:

# I want to buy a refrigerator,. Which platform is good and cheap?

# I want to buy a plane ticket,. Which platform's ticket time is right and the price is cheap?

# I want to rent a house, Lianjia, I love my family, Anjuke. Which intermediary provides the highest price-to-service ratio?

When the user hesitated, the crawler came on. It crawled away the messages of each family and moved other people's cheese.

Second, play: the resources of "reshuffle and reorganization"

In the era of national fitness, most people think that the annual card in the gym is very cost-effective, but only 10 days a year after the card is in hand. Originally calculated that so many equipment in the gym only cost 6 yuan per day (~ = 2200 yuan / 365days), but the cost has become 200 yuan / time, inexplicably increased 33 times. Now think about it, in fact, the gym is to use the "high idle rate of resources" to make money, well, fitness is very difficult to persist.

Then how to use the transformation of these idle resources to make money, then it is time for reptiles to show their value.

Take the shared bike in the tuyere as an example, the shared bike in the streets and alleys are colorful, each color represents a company, and they want to stand in this tuyere and be the pig. The industry model is more or less the same, registration-deposit-scan code-cycling-billing, the difference for users is only in the amount of deposit, whether there are many cars in the activity area, single-time and monthly / quarterly / annual card billing method. However, after users have registered on various platforms, they use to find out:

# Why are there so few bikes nearby where I pay the deposit? I can't ride another bike in front of me.

# A deposit of several hundred is a bit expensive. What if you can't ride it if you don't pay it?

Unluckily, someone made a platform using crawler technology to integrate all the mainstream shared bike information. Users do not need to pay a deposit after registration, as long as a very small monthly fee, you can scan any bike to unlock and ride away. Why can you ride away? Because the crawler integrates your ownership of monthly / quarterly / annual cards on all shared bikes, opening up your corresponding use rights after scanning each bike. In this way, it not only diverts users from other platforms, but also earns the income of other people's "idle resources" after the reorganization of resources. It does meet the needs of users, but it also moves other people's cheese.

two。 The intelligence war caused by reptiles

There is a saying in Sun Tzu's Art of War: "if you know yourself and your enemy, you will not be defeated in a hundred battles." Some companies also use reptiles to obtain peer intelligence to fight. Take stock of the three most popular campaign intelligence in the past two years, that is, the crawler's favorite information: the merchants of the O2O war, the price of the e-commerce war, and the risk control of the P2P war.

For O2O platform to be successful, more high-quality merchants need to be introduced to attract more C-end users, and the whole business model can be changed. Therefore, the industry crawls each other's high-quality shops with high sales.

Reptiles continue to challenge the "bottom line"

The other side of the reptile is reminiscent of Liu Jianming, played by Liu Dehua in Infernal Affairs, who works undercover in the police station and works for the gang on that red line. Black ash products use crawler technology to step on the edge of the law to make a lot of money, evil scenes everywhere, the two are the same.

●, the high-scoring store you see when you open a review is likely to be judged by a reptile.

●, the popular styles you see on major e-commerce platforms may also be fake transactions of reptile brushes.

The first place in the ● vote may be the result of the hard work of the crawler.

The Weibo Big V where it is easy for ● to make money may be the illusion that many zombie fans made by reptiles stand up.

The first place in the program promotion and installation rewarded by ● channel merchants may be due to the hard work of the crawler.

Customers of ● e-commerce platform can never get red packets, because the crawler has robbed you of all the money.

●...

In addition, because reptiles tend to move faster than people, they also engage in hoarding. Tickets for Jay Chou's concert are hard to get, but scalpers are selling at a high price at the door of the concert with a stack of tickets, and fans can only endure the pain to see their idols at three times the price. Such resale profits are very common. Crawlers often stare at the Internet to buy high-selling items at a low price and make a profit from the price difference, such as cheap air tickets from AirAsia, train tickets for the Spring Festival, second-priced Maotai on the website, and expert reservation numbers from well-known hospitals. Under his use, such reptiles grabbed resources and disrupted the market.

There is nothing wrong with reptiles, but in the final analysis, it is man-made. We can't destroy reptiles just because someone uses them to do evil, but we can't allow malicious reptiles to grow wantonly, so a long-term struggle between reptiles and anti-reptiles begins. Ali Yundun reptile risk management product is an inevitable product. This product can manage the reptiles in an orderly way, prevent them from doing evil and carry forward their good.

The product has built a set of hierarchical crawler detection modules, which are mainly divided into basic protection layer, cloud intelligence layer and deep learning layer, just as every police station and gang has a complete set of information transmission and identification mechanism. from providing custom crawler feature tools on traffic, to industry crawler attack intelligence sharing cloud advantages, to customized depth learning algorithms for customer business, step by step. Quickly help customers to create a tailor-made anti-crawler strategy system.

Cdn.com/e28c18195578d97d65218546956849a8b5329fb4.png ">

The changeability of reptiles needs to be identified. Cloud Shield reptile risk management products can quickly identify the behavioral characteristics of reptiles, do different treatments for different reptiles, release them, kill them, and make a challenge or verification when they encounter suspicious reptiles. one more physical check to make the final judgment.

At the same time, it also provides data visualization, showing users from different dimensions, including the relationship between data and data, allowing users to participate in the exploration of the relationship between crawler characteristic data and protection data, and constantly communicate and iterate. not only clearly show users every step of crawler invasion, but also improve the user's anti-crawler strategy decision-making ability.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.