Shulou(Shulou.com)06/03 Report--

Isn't it a great experience that when you drive past a restaurant's parking lot, the restaurant's specials of the day pop up on your mobile phone screen? Would you be a little excited if the boss gave you back the $20 that the dealer forgot to pay you? Would the world be a wonderful place if online video games could immediately tell us about users who are similar to us? Are you going to lower the car insurance rate? Big data can make all this come true.

Even if network data is not the most original large data source, it is also the most widely used and most recognized large data source. In addition, there are many large data sources, all of which have their own use value. Some of them are well known, while others are almost unknown. We would like to borrow the space of this chapter to review the other nine large data sources in addition to network data and their uses. We will explain this part at a high level, with the intention of reviewing the application and commercial meaning of each large data source on the basis of a brief description of all kinds of data sources.

We have found a very obvious trend that although various industries have generated many large data sources, the underlying supporting technologies are the same. Moreover, different industries can use the same large data source. Big data is not only a single use, its impact will be very far-reaching.

We are going to discuss the following sources of big data

Auto insurance: the value of on-board information service data.

Multiple industries: the value of text data.

Multiple industries: the value of time data and location data.

Retail manufacturing: the value of RFID data.

Power industry: the value of smart grid data.

* * Industry: the value of data tracked by chips.

Industrial engines and equipment: the value of sensor data.

Video games: the value of telemetry data.

Telecommunications and other industries: the value of social networking data.

Auto Insurance Industry: the value of on-board Information Service data

The attention of in-vehicle information service in the automobile insurance industry is very high. The on-board information service collects and grasps the relevant information of the vehicle through the built-in sensors and black boxes. We can configure different schemes and use black boxes to monitor all car data. We can monitor the speed, mileage, and whether the car has an emergency braking system. Vehicle information service data can help insurance companies better understand the risk level of their customers and set reasonable insurance rates. If privacy issues are completely ignored, the in-car information service can track all the places the car has been to, when it has arrived, how fast, what functions of the car have been used, and so on.

In-car information service can potentially reduce the premium rate for drivers and increase the income of insurance companies. How does it increase revenue while lowering rates? The answer is that insurance companies should price insurance according to risk assessment. Traditional risk assessment methods use data such as age, demographic characteristics and personal accidental injury history, which can only provide high-level summary information. For car owners who have no problems with their driving records, traditional methods simply cannot distinguish them from other people in the neighborhood.

Insurance companies should prepare in advance and prepare for the worst. They need to figure out which people are the safest in which risk range, and in general, they will assume that their risk is at the higher end of the risk range. The more detailed the car insurance company knows about the owner's behavior and actual risk, the narrower the risk range will be, and the less likely it will be that a worst-case scenario in which premium rates need to be raised. This is why it is possible to lower insurance rates and increase returns at the same time. If the insurance company thinks that the risk of the insured individual is better, then the insurance company will be able to better understand everyone's risk situation, and it is expected that the premium that must be paid will not change much.

Insurance companies in many countries around the world are using on-board information services, and the number is increasing. Early projects focused on gathering the least information from cars, for example, they didn't care where the car had been. Early projects tracked how far the car drove, when it was driven, whether it was speeding and whether a large number of emergency brakes were used. This information is very basic information, does not involve personal privacy, is deliberately designed like this. It is widely accepted because it avoids the collection of highly sensitive information. The same applies to commercial fleets. If the insurance company knows more about the use of its fleet, it will be easier for it to set insurance rates for the company's fleet.

Vehicle information service data first appeared as a tool to help car owners and companies get better and more effective vehicle insurance. After a while, when many vehicles are equipped with in-vehicle information service devices, industries outside the insurance industry can also use in-vehicle information service data. Now, the bus has an on-board computer management system, but on-board information service equipment can take it to a new level. There are also some interesting applications for in-car information service data. Let's take a look at these applications.

Use on-board information service data

If the on-board information service really starts to be used on a large scale, there will be many exciting analytical applications. Imagine that tens of millions of cars across the country will be equipped with on-board information service devices, and third-party research companies will collect very detailed vehicle communication data for customers anonymously. Unlike the limited data collected for insurance, data collection is in minutes or seconds, and the collection includes, but is not limited to, speed, location, direction, and other useful information.

No matter whether the traffic is stuck or not, no matter what date it is, this kind of data feedback will provide a lot of vehicle communication information. Researchers can know the speed of each car on the road, and they can also know when the traffic starts, when it ends, and how long it lasts. How amazing this real view of traffic flow information will be! Imagine how much impact this will have on the study of traffic congestion and road system planning!

Inadvertently insert willow into yin

The multiple uses of in-vehicle information service data is just one example of how big data can be used in ways that were initially unforeseen. For a particular data source, we finally found that its most effective use may be very different from its original use. In the face of every kind of large data source we encounter, we should open up our minds and think more about other uses other than conventional ones.

If researchers can grasp the movements of a large number of cars in every peak hour, every day, and every city, they can clearly determine the causes and consequences of traffic flow. In addition, the answers to the following questions can be found.

What is the impact of a tire in the middle of the road on traffic?

What happens to traffic jams in the left lane?

What will happen if the traffic lights at the intersection are out of sync?

Which intersections work as expected, but the design of travel time is still unreasonable?

If one road is blocked, how quickly will the congestion spread to other roads?

Even if we concentrate on expensive tests, it is almost impossible to study such problems effectively. Unless we arrange people to actually monitor every road and record all the information, only in this way can we solve the problem of traffic jams. Alternatively, we can install a large number of sensors to monitor passing vehicles and video cameras, but these options are severely limited because of cost.

Traffic road engineers dream of getting what we call in-car communications. If in-car communication devices become ubiquitous, any traffic jam can be found. The innovation of urban road and traffic management systems, as well as urban road construction planning, will benefit the general public. When vehicle communication first appeared, it was to meet the demand of insurance pricing, but it can also relieve traffic pressure and drivers' anxious waiting when they are in traffic jams. Its existence will eventually revolutionize the management mode of expressways.

Multiple industries: the value of text data

Text is one of the largest and most common large data sources. Think about how many text messages exist around us, e-mails, text messages, Weibo, posts on social media sites, instant messaging, real-time meetings, and recorded messages that can be converted into text. Text data is now the least structured, but also the largest large data source. Fortunately, we have done a lot of work in harnessing text data and using text data to make better business decisions.

Text analysis generally starts with parsing the text, and then assigns semantics to various words, phrases, and parts that contain the text. We can do text analysis through simple word frequency statistics or more complex operations. There are already a lot of such analyses in natural language processing, so we won't repeat them here. Text mining tools are an indispensable part of mainstream analysis suites. In addition, we can find many independent text mining toolkits. Some of these text analysis tools use rule-based methods, and users need to adjust the software to find the patterns they are interested in. Other tools use machine learning and other algorithms to automatically discover data patterns. Each method has its own advantages and disadvantages, and its related discussion is beyond the scope of this book. We are concerned with how to use the generated results, not the process of using tools to produce results.

After parsing and classifying the text, we can analyze the results of these processes. The output of the text mining process is usually the input of other analysis processes. For example, if you can analyze the emotion of a customer using e-mail, you can use a variable to mark the customer's emotion as positive or negative. The tag itself is a structured data that can be used as input to the analysis process. Create structured data using unstructured text, a process often referred to as information extraction.

Another example is that assuming that we can identify their evaluations of some of the company's products in the emails that customers communicate with the company, we can use a series of variables to identify customers' product evaluations. These variables themselves are structured metrics and can be used for analysis. These examples explain how to capture unstructured data fragments and extract relevant structured data from them.

Extract structural data from unstructured text

The example of text analysis is a good illustration of the process: getting unstructured data, then processing it, and finally creating structured data that can be used in the analysis and reporting process. An important part of driving big data is to use this creative way to turn unstructured and semi-structured data into data that can be analyzed.

Interpreting textual data is actually quite difficult. If the emphasized words and context are different, the meaning of the same word is different. In the face of pure text, we simply do not know what the point is, nor do we know the whole context. This means that we have to make some assumptions in advance, which we will discuss in more detail in Chapter 6.

Text analysis is not only an art, but also a science, there is always some uncertainty. Text analysis often has the problems of misclassification and ambiguity. Yes, if we find a better decision support pattern in the text collection, we should use it. The goal of text analysis is to improve your decisions, but not to make them perfect. Text data can effectively improve the effectiveness of decision-making, and it can provide better results than without it, even when the data is noisy or ambiguous.

Using text data

One of the most popular text analysis applications is the so-called emotional analysis. Emotional analysis is to dig out the overall point of view from a large number of people and provide market comments, views and feelings about a company and other related information. Emotional analysis usually uses data from social media sites. Here are a few examples of emotional analysis.

What is the reputation of the company or product?

What activities of the company are we talking about?

Is everyone's evaluation of the company, products and services good or bad?

As mentioned earlier, the difficulty of text analysis is that vocabulary and context are related. We have to consider this problem, but a large number of comments will make the customer's emotional tendency clear. If we can interpret the trend of what people say on social media and interact with customer service, it will be of great value in planning the next step.

If the company can grasp the emotional information of each customer, it can understand the customer's intention and attitude. Similar to the method of using network data to infer a customer's intention, it is also valuable information to know whether the customer's overall emotion towards a product is positive or negative. If the customer has not purchased the product at this time, it will be even more valuable. The information provided by emotional analysis can let us know how easy it is to persuade the customer to buy the product.

Another use of text data is pattern recognition. We sort customer complaints, maintenance records and other evaluations in the hope that problems can be identified and corrected more quickly before they get bigger. The product is released for the first time, and then complaints begin to appear, and text analysis can identify where the customer has problems. We can even identify the problem before customer service calls come in one after another. In this way, we can respond more quickly and positively. Companies can respond in a timely manner to solve the same problems in future releases of the product, and can also take the initiative to contact customers to ease their anxiety when they encounter difficulties.

Fraud detection is also one of the important applications of text data. In the case of health insurance or disability insurance complaints, text analysis technology can be used to analyze customer comments and reasons. Text analysis can identify fraud patterns and mark the level of risk. In the face of high-risk complaints, they need to be examined more carefully. On the other hand, complaints can be executed automatically to some extent. If the system finds that there is nothing wrong with complaint patterns, words and phrases, it can determine that these complaints are low-risk and can speed up processing, while devoting more resources to high-risk complaints.

Legal affairs will also benefit from text analysis. It is customary for any legal case to request corresponding e-mail and other communication history before appealing. The text of these communications will be checked in batches to identify the statements relevant to the case. For example, which emails have hidden inside messages? Which people are telling lies when communicating with others? What is the essence behind the threat?

The application of text analysis in legal cases is called electronic reconnaissance. All advance analyses will contribute to the success of the prosecution. Without text analysis, you will not be able to browse all the required documents manually. Even if we can browse those documents manually, because the task itself is too monotonous, we are likely to miss some of the key information.

Text data may have an impact on all industries. It is probably the most widely used class of big data today. It is important for enterprises to master how to collect, parse, and analyze texts. Text is a big data source that we must control.

Multiple industries: the value of time data and location data

With the advent of Global Positioning system (GPS), personal GPS devices and mobile phones, time and location information has been increasing. From Foursquare to Google Places to Facebook Places, they provide a large number of services and applications that can record everyone's location at a certain point in time. Mobile apps can record our location and movement. Even if the phone does not officially turn on GPS, we can still use the base station signal to get quite accurate location information.

There are some novel ways to use this information in consumer applications that capture the information that consumers allow them to capture. For example, there are apps that allow us to track the route during exercise, the length of the route, and the time it takes to complete the route. In fact, if we carry a cell phone, we can record every place we have been. We can also choose to disclose the data to others. When more people disclose their time and location data to the public, some very interesting things will happen.

Many companies have begun to realize the power of mastering customer time and location data, and they have begun to try to collect this kind of information from customers. Of course, such information must be based on screening, and must have clear privacy policies and strict compliance with these policies. Many companies have launched irresistible location value services that attract users to open up time and location information to them.

We don't just want to know the time and location of consumers. The leader of the truck fleet also wants to know the location of each truck at a certain point in time. The pizza parlor must want to know where each delivery person is at a certain time, and the pet owner must want to know where the pet is outside. Organizers at large banquets need to know how efficiently waiters move around and how quickly they respond to customers.

Starting from collecting data on the time and location of individuals and assets, enterprises can quickly enter big data's field. It would be better if this information could be updated frequently. It is one thing to know where each truck is in the morning and evening, and another to know where each truck is every second. Time and location data will be adopted and applied to a higher and higher degree, and its impact will be greater and greater.

Use time and location data

Time and location data are the most sensitive to privacy of big data. We are faced with not only privacy issues, but also moral and ethical issues. Should we put chips on children's arms so that they can be tracked when they get lost? What should we do when Alzheimer's patients run away from home or leave care institutions without permission? Of course, the possibility of time and location data being abused is quite high. But on the bright side, there is also a high chance that they will be used properly. Let's take a look at some examples.

People may soon register with police stations and fire departments and provide information about where they go on a daily basis. In this way, in the event of a major event such as a flood, fire or road closure, people will receive warnings from police and fire departments telling them that there is a situation in the area they are about to pass by and reminding them to make a detour. If people can take the initiative to avoid right and wrong, the time of traffic interruption can be minimized, so that everyone's time can be saved. Finally, with your permission, the local government can even receive your real-time location information.

One incipient way to use data is to develop message notifications that are sensitive to time and location information, and there is plenty of room for the future of this market. Notifications are no longer limited to the same day or this week, but provide the most appropriate message notifications based on the customer's time and location information. The current practice is for customers to check in and tell them where they are, so that they can receive notification messages. The company can continuously track the movements of customers in order to respond accordingly.

For example, a user might tell you that he wants to leave the office and go home at 5:30 and will drive through exit 5 between 5:45 and 6:00. He wants to find a place to eat and wants to know what food is available in your store or restaurant at that time. You need to provide delicious meals that match his needs at that time and place. It was obviously too late to tell him the relevant information by email the next morning, and what we wanted was to send him the notification message the moment he passed through the place.

Actively push notification information according to place and time

A growing trend in the field of marketing is to push notification messages only to customers who just benefit from a certain period of time and a certain location. Compared with notifications sent based on a wide range of time and place, such notifications are more effective and targeted. Early companies that adopted this approach have achieved surprising results.

Of course, managing such notifications is much more complex, because we need to do more than just track everyone's weekly service recommendations. What we need to care about is where each user is all the time, and what we recommend to them at this point in time. Pushing notifications based on time and location does add a lot of complexity and become difficult to manage. But we believe that over time, if we do well, the conversion rate in this way should far exceed the traditional personalized recommendation. Historical experience tells us repeatedly that the more accurate the notification information is, the higher the conversion rate will be.

Another model for using such data is enhanced social network analytics. Wireless operators can identify the relationship between users based on voice and text communication information, and use time and location data to identify who are in the same place at the same time. For example, who is listening to a concert or watching a movie? Who is going to watch a certain sports match? Who eats in the same restaurant at the same time?

If you can identify people who show up at about the same time and place, you can identify people who don't know each other or are in the same social circle, but they all have a lot in common. Imagine if matchmaking services could use this information to help us find our partner. We can encourage people to make connections and provide them with product recommendations that match their personal or group identity.

Time and location data can not only help us understand customers' historical patterns, but also accurately predict where customers will appear in the future. This is especially true for customers who have fixed habits. If we know where someone is going and where they are going, we can predict where they will be in 10 minutes or an hour. If we know where the customer has been on the same road before, we can make a more accurate prediction of where he is going now. At worst, we can greatly reduce the candidate routes on the list, so that we can support more accurate marketing.

In the next few years, the use of time and location data will experience explosive growth, and consumer-oriented selection processes and incentives will eventually mature. Now we have to be careful and get the user's permission before we use this information. Message notifications that use time and location data will be more targeted and personalized. In the near future, if the notification information is not pushed based on time and location, it may be considered corny.

Retail Manufacturing: the value of RFID data

Radio frequency tag, namely RFID tag, is a kind of miniature tag installed on the shipping pallet or the outer package of the product. The RFID tag has a unique serial number, which is different from the generic product identification code similar to UPC. In other words, the RFID tag can identify not only the Model 123s on the trays, but also a unique and specific set of Model 123s loaded on the pallets.

The RFID reader signals, and the RFID tag returns response information. If multiple tags are within the range of the card reader, they will also respond to the same query, making it easier to identify a large number of items. Even when these things are stacked together or placed behind the wall, as long as the signal can penetrate, we can get the response information. With the RFID tag, we no longer need to manually record and count every item, so the time to count goods will be shortened.

Most RFID tags used outside of high-value applications are passive tags, meaning they don't have built-in batteries. The radio waves of the card reader generate a magnetic field that provides enough energy for the tag to send out the built-in information. RFID technology has been around for a long time, but the cost limits the further promotion of the application. Today, the cost of passive tags is only a few cents, and the price is falling. With the continuous decline of the order, the actual application will continue to grow. There are still some problems with the current RFID technology, for example, the liquid will block the signal of the label. With the passage of time, these technical problems will be solved effectively.

There are some RFID apps that many people have come into contact with, one of which is the automatic toll tag. With it, drivers don't have to stop when they pass through the highway toll station. It works by inserting a RFID tag into the card issued by the Transportation Administration and a card reader on the highway; when the car passes, the tag sends the car data to the card reader so that we can be recorded when we drive through the toll booth.

Another important application of RFID data is asset tracking. For example, a company wants to label every PC, desk, chair, TV and other assets it owns. These labels can help us with inventory tracking. Track these items. If items are moved out of the specified area, they will send a warning message. For example, we can put the card reader at the exit, and if the company's assets go out without prior approval, the alarm will soon go off, which can serve as a security warning. This is similar to the label of an item in a retail store. If the label becomes invalid, the alarm will be set off.

One of the biggest applications of RFID is pallet tracking in manufacturing and item tracking in retail. For example, there is a label on every tray that the manufacturer sends to the retailer, so that you can easily record which goods are in a distribution center or store. Eventually, low-priced items in stores can also be equipped with RFID chips, or use a similar new technology. Now that we understand what RFID data is, let's take a look at how RFID data can improve the current business model.

Use RF tag data

One of the value-added applications of RFID is to identify whether there is a corresponding item on the retailer's shelf. If the card reader can continuously determine the stock of each item on the shelf, we can get accurate information when it needs to be redistributed. Using RFID can better track the supply status of shelves, because the status of goods out of stock is completely different from that in which goods are available. One possibility is that there are no more items on the store shelves, but there are still five items in the back storeroom.

In this case, any traditional out-of-stock analysis will show that there is still stock on the shelf, so there is no need to worry. People find out where the problem lies when sales start to decline. If you have a RFID tag, you can trace that there are five more items in the storeroom, but there are no such items on the shelf. In this way, we simply need to move the goods from the storeroom to the shelf to solve the problem. There are some cost and technical challenges in this example, but now we are trying to overcome these difficulties.

RFID can also help us track the impact of promotional displays. Usually in the promotion process, the goods are displayed in many locations of the store. From the traditional POS data, we can know the sales volume of the promoted goods, but we don't know which display point the sales come from. Through the RFID tag, we can identify the display point from which the goods are sold, so that we can evaluate the impact of different locations on sales results.

RFID can be more powerful when combined with other data. If the company can collect temperature data in the distribution center, we can track the damage to the goods in the event of a power outage or other extreme events. Maybe the temperature in an area of the warehouse is as high as 90 degrees Celsius during a power outage and the time is as long as 90 minutes. With RFID, we can know exactly which pallets are located in that area of the distribution center at that moment, and then we can take action accordingly. The warehouse data can also be matched with the shipping data, and if the goods are damaged, the company can recall the goods and notify the retailer to check the goods again when they arrive.

The combination shows magical powers.

Like many other big data sources, RFID data itself does not exert all its power. They work when used in combination with other data. The goal of big data's strategy is to integrate big data and other data into the same process, which cannot be overemphasized. Using big data is not an isolated job.

RFID also has some operational applications. The commodity management of some distribution centers is not strict, resulting in a high degree of damage to the goods. This is true for some teams, even for some workers. The Human Resources (HR) system reports who works at any point in time. When combined with such data, RFID data can show when the goods have been moved and identify employees who are more likely to damage, wear and tear, and steal goods. The combined use of data enables us to take stronger and higher-quality actions.

RFID has a very interesting future app for tracking store shopping activities, just like tracking Web shopping behavior. If the RFID card reader is embedded in the shopping cart, we can know exactly which customers put what in the shopping cart, as well as the order in which they put it. Even if not every item is labeled, we can still identify the road the shopping cart passes through. Many of the benefits that can be achieved by using RFID,Web data in the storefront will become a reality. Privacy must be considered in the last two examples, because customers may not want their shopping behavior to be tracked at all. We can use the method of "anonymous" shopping without identifying the location of the person who generated the data.

The final application of RFID is to identify fraudulent criminal activities and return stolen goods. If the item is labeled with RFID, the retailer can identify it through the ID of the label, determine whether the returned item belongs to the same batch of stolen products, and take appropriate action. In fact, the key is that RFID's ID can be used as part of the receipt to assist in the return process. Retailers know which RFID label is affixed to the purchase, rather than just knowing that you have purchased something as usual. When we come to the return desk, we want to return the goods with that label. We certainly can't take another identical item off the shelf and pretend to return it with the receipt. Using RFID in this way, fraud will become extremely difficult.

RFID is likely to have a huge impact on manufacturing and retail in the next few years. Contrary to many people's expectations, the acceptance of RFID is slower. However, the price of RFID tags continues to fall, while the quality of tags and card readers continues to rise. From an economic point of view, RFID will be more widely used.

Power industry: the value of smart grid data

Smart grid is the next generation of power infrastructure. Compared with the high-voltage transmission that we often see around us, the smart grid is more advanced and reliable. The smart grid has very complex monitoring, communication and power generation systems, which can provide stable services and recover better and faster in the event of power outages and other problems. Various sensors and monitoring devices record a lot of information about the power grid itself and the current flowing through it.

One part of the smart grid is the smart meter that we often talk about. Intelligent electric meter is a substitute for traditional electric meter. From the outside, the smart meter is no different from the meter we have been using, but the function of the smart meter is more powerful. In the past, meter readers used to read electricity meters from house to house every few weeks or months, while smart meters could automatically collect data from every household or enterprise every 15 minutes to an hour, or even across regions or power grids.

Although we focus on smart meters here, the sensors that are widely used in the smart grid are also worth mentioning. The data collected by these sensors, which are all over the smart grid but we can't see, dwarf the smart meter data in terms of scale. The sensors read 60 synchronous phasor measurements from the power generation system per second, just like the home network that records the switching status of household appliances, they are the examples of big data. Ordinary people do not know that these sensors exist, but they are very important to the power grid. The sensor has to read all the current data and the equipment status of the smart grid, and the amount of data is very large.

Smart grid technology is already in use in some parts of Europe and America. We believe that in the near future, every power grid in the world will be replaced by a smart grid. Because power companies use smart grids, the amount of power consumption data they have will grow exponentially. How to use this kind of data? Let's take a look.

Using smart grid data

From the point of view of power consumption management, smart meter data can help people better understand the needs of customers in the power grid. In addition, these data can also benefit consumers. For example, the owner can choose to turn on the appliances to be tested while keeping other appliances stable, and the detailed power consumption can be monitored from the smart meter. In this way, we can clearly measure how much electricity is consumed by various appliances.

Power companies around the world are now actively turning to such pricing models, that is, pricing according to changes in time or demand, and the emergence of smart grids has accelerated this trend. One of the main goals of power companies is to use new pricing procedures to influence customer behavior and reduce electricity consumption during peak hours. In order to cope with the peak power consumption, another power station needs to be built, which requires a lot of money and will have a great impact on the environment. If the cost of electricity can be flexibly set according to time and measured by smart meters, we can urge customers to change their electricity behavior. Lower peaks and more stable electricity demand equate to less demand for new infrastructure and lower costs.

Of course, power companies can identify other trends through the data provided by smart meters. Where has the electricity consumption come down? Which consumers have the same daily or weekly electricity demand? Power companies can classify customers according to usage patterns and can choose to develop products and activities for specific groups. Using this data, we can also identify where the pattern is abnormal, which reveals the problems that need to be solved.

In fact, power companies have the ability to perform customer analysis work that has been used in other industries for many years. For example, the phone company knows all our bills at the end of the month, but it doesn't know exactly what we're talking about. Retail stores only know the overall sales situation, but do not know any details of the purchase. A financial institution knows our month-end balance, but it doesn't know our capital flow this month. In many ways, this kind of data faced by power companies is still slightly inadequate for understanding customers. They also have simple month-end summary data, but such monthly data are often estimates rather than actual power consumption.

Big data can change an industry.

Sometimes, big data can really change an industry and elevate analytical applications to a whole new level. The smart grid data used by the power industry is one such example. No longer subject to the restriction of monthly meter reading, power consumption information will be measured in seconds or minutes. The ingenious sensors all over the power grid make the use of data completely different from before. The data analysis based on this will produce a lot of innovation in many aspects, such as rate package, electricity management and so on.

With smart meter data, we can do a whole new analysis that will benefit all the public. Consumers can customize rate packages according to their own usage patterns, just as in-car information services support personalized car insurance rates. The charges of electricity customers during peak hours are higher than those during off-peak hours. In the face of such a stimulus, we will change our electricity consumption pattern, and maybe we will use the dishwasher later in the afternoon instead of immediately after lunch.

Power companies will also have more accurate demand forecasts, and they will be able to identify more clearly where the demand comes from. They can also understand the electricity needs of certain types of customers at a certain time. Power companies can use different methods to drive various behaviors, make demand more stable, and reduce the frequency of abnormal peak demand. All this will dampen the demand for expensive new power generation equipment.

Every household and every industry can feel the power of smart meter data, which allows us to better track and actively manage electricity consumption. We can not only save electricity, but also make the world more low-carbon, but also help people save money. If we can clearly know that we consume more electricity than expected, we will certainly make appropriate adjustments according to our needs. If we only use monthly bills, we will not be able to recognize this opportunity. But smart meter data will make it all easier.

* * Industry: the value of chip tracking data

Earlier we have discussed how RFID technology is applied to retail and manufacturing. RFID technology is actually more widely used, and many applications will produce big data. Another application of RFID tags is affixed to chips used by × ×. Each chip, especially the high-value chip, has its own built-in tag, so that it can be uniquely identified by the serial number of the label.

The xx used in xxx has been tracked for many years. Once we swipe a frequently used player card or credit card on xxx, every time we move the handle and press the button, we will be tracked. Of course, your bets and the money you win will also be followed. Although the analysis of × × mode has a long history, it still doesn't capture enough details from desktop games. Now that process is changing, tags have begun to be implanted into the game chips.

In the past, × × would use a powerful security camera to track the chips, and the ground crew's job was to ensure that the chips moved up and down reasonably. Table managers look for regulars, estimate their average betting and playing time, and reward such regulars. Although gaming table managers are good at this and can get help from other people, game rewards are always more or less inaccurate. This inaccuracy can occur if the monitored player happens to bet a little more or less than usual. If some players think they are being watched, they will use the rules of the system to increase bets to make a profit.

Similar technologies can drive a variety of × × ×

Both retailers and manufacturers use RFID technology. The same is true of the industry. There are many differences in the way they use RFID, but there are also many similarities. The most interesting thing is that a technology can be used in different industries to form a unique large data source for each industry.

Chip tracking is a special RFID application, in addition to this example, RFID has many other applications. This example shows that some of the same underlying technologies can support different xxx. these xxx are essentially the same, but the scope and application are completely different. What excites us is that this basic technology has completely different uses, resulting in different forms of big data in a variety of industries.

Industrial engines and equipment: the value of Sensor data

Many complex machines and engines have been installed all over the world, such as aircraft, trains, military vehicles, construction equipment, drilling equipment and so on. Because of the high cost, it is very important to maintain the stable operation of these equipment. In recent years, embedded sensors have been used on a variety of machines, from aircraft engines to tanks, with the goal of monitoring the status of equipment in seconds or milliseconds.

Monitoring can be done in considerable detail, especially during testing and development. For example, when a new engine is developed, it has to rely on enough detailed information to check whether the engine can work as expected. Once the new engine enters the market, the cost of replacing defective parts will be quite high, so we need to conduct a detailed performance analysis in advance. Monitoring is an ongoing activity. Maybe we don't need to collect every millisecond details continuously, but if we can collect a lot of details, we can evaluate the life cycle of the device and identify repeated problems.

For example, the engine sensor can collect information from temperature to revolutions per minute, fuel intake rate, and oil pressure level, and the data can be obtained at a predetermined frequency. When the reading frequency, the number of indicators and the number of monitoring items increase, the amount of data will increase rapidly. Why should we care about that? Let's take a look at some examples.

Using sensor data

The structure of the engine is very complex, there are many moving parts, must run at high temperature, will go through a variety of operating conditions. Because their costs are too high, the longer the life expectancy, the better. Therefore, stable and predictable performance becomes extremely important because the life of the machine depends on it. For example, maintaining a faulty aircraft can cost the airline or the air force a lot of money, but we still have to do this because we need to identify whether there is a safety hazard. Therefore, the downtime of aircraft or aircraft engines and other equipment must be reduced to a minimum, which is urgently needed by airlines or air force units.

Downtime minimization strategies include preparing spare parts or equipment that need to be repaired when backup engines are quickly cut, quickly identifying parts that need to be replaced from diagnostic results, and investing in new and more reliable versions of faulty parts. In order to implement these three strategies effectively, there must be data. We need to use data generation diagnostic algorithms, or data as input to diagnose a particular problem. The engineering department can use sensor data to accurately locate the cause of the problem and design new measures to support longer and more reliable operations. These considerations apply whether the engine is from an aircraft, a ship, or land equipment.

By extracting and analyzing detailed engine operation data, we can accurately locate certain modes that can cause immediate failure. Then we can identify time segmentation patterns that reduce engine life and more frequent maintenance. The number of permutations and combinations of multiple variables, especially over a period of time, makes this kind of data analysis a challenge. This process will not only involve big data, but also the analysis developed with it will become extremely complex and difficult. Here are some questions that we can study.

Does a sudden drop in pressure mean that something will go wrong?

Does the continuous drop in temperature within a few hours mean that there are other problems?

Does the abnormal vibration level mean that there is a problem?

Does the rapid rotation of the engine during startup seriously damage the performance of some parts and increase the number of repairs?

The oil pressure has been low for several months. Will some parts of the engine be damaged?

Lack of structure in structured data

Sensor data brings us a very difficult challenge. Although the data we collect is structured and independent data elements are easy to understand, the temporal relationships and patterns between elements are simply incomprehensible. Delays and unmeasurable external factors add to the complexity of the problem. If you want to consider all the information and identify the long-term effects of all kinds of data, the process can be extremely complex. Having structured data does not necessarily guarantee that the analysis method is highly structured and standardized.

When there is a serious problem, it can be very effective to go back and check what happened until the problem shows itself. The function of the sensor is similar to relying on the help of aircraft black boxes to diagnose the cause of the crash. Engine sensor data can be used to diagnose activity and research behavior. Conceptually, the sensor we are talking about here is a more complex form than the information service equipment in the car insurance case we talked about earlier. Sensors constantly perceive the surrounding environment and obtain data information, which is a topic repeatedly discussed in big data's world. Although we are talking about engines here, sensors have countless uses, and the principles discussed here also apply.

If a large number of sensors repeat the sensor data collection process for a long time, it will produce a large amount of rich analysis data. As long as a good analysis of these data, we can find the defects of the equipment, and have the opportunity to take the initiative to fix these problems. We can also identify the weaknesses in the equipment first. Then we can develop a process to alleviate the problems caused by these findings. The benefits of these measures will not only increase the level of security, but also reduce our costs. Using sensor data, the engine and equipment will be safer, and the time it takes to provide services will be longer, so the operation will be more stable and the cost will be lower. This is a win-win approach.

Video games: the value of telemetry data

Telemetry data is a term used in the video game industry to describe the state of capture game activities. Its concept is no different from the network big data we talked about in Chapter 2, because telemetry data collects the activities of players in the game. Telemetry data are collected mostly from online games rather than palmtop games.

In a hockey game, telemetry data collect when the player scores the ball, which method of hitting the ball, and how fast the ball is. In war games, telemetry data collect what kind of fire is used, where the fire is opened, in which direction the fire is fired, and the degree of damage to various things. In theory, all the details of the relevant scenes and activities can be collected.

Video game manufacturers can easily see not only how many customers have bought game software, but also how many hours the game has been played. Using telemetry data, game makers can learn about their customers' private information, how they actually play, and how they interact with the games they create. The game data we collect may be large, but the video game industry has begun to actively analyze the data. Telemetry data have an impact on many areas. From the advantages and uses of telemetry data, it is easy to find the similarity between telemetry data and network data. Let's take a look at some examples.

Use telemetry data

Many games make money from the subscription model, so maintaining refresh rates is important for these games. By mining the player's game mode, we can know which game behaviors are related to the refresh rate and which are irrelevant. For example, perhaps in sports games, the use of some auxiliary features will greatly increase the refresh rate. Game makers will take steps to attract players to try the game, in order to induce them to use features that have not been used before.

The telemetry data will only get bigger and bigger.

Nowadays, most of the objects captured by telemetry data are control handles or keyboard behavior. With the development of interactive games, they can track the movements of players instead of relying on control handles, resulting in a surge in the amount of data. Knowing what button a player presses at what time is much less than knowing the spatial position, direction and speed of a part of his body at a given moment.

Newer games tend to allow players to spend a little money on items during the game, which is called microtransaction. For example, a special weapon sells for only 10 cents. We can analyze the game and identify where the success rate of such micro-transactions is higher. Maybe somewhere in the game provides a very handy weapon that will cause a frenzy among players. We can use the quick prompt on the screen to tell the player that there is a weapon to buy now, so that many players will choose to buy the weapon.

Like other industries, customer satisfaction is also a big problem in the video game industry. The uniqueness of video games lies in setting a very wonderful route. The game should provide players with challenges, but the challenges should not be excessive. Excessive challenges will make players feel frustrated and give up the game. If the game is too simple or too complex, players will get bored and turn to other games.

Through game analysis, we can identify which levels in the game each player can easily pass, and which levels are difficult for even the top players. We can increase or decrease the number of enemies in these places and try to make the difficulty level more balanced. A balanced difficulty level of the game can provide players with a more consistent experience and make them more satisfied. This will lead to higher refresh rates and more buying behavior.

Through telemetry data, players can also classify according to the game style. Using this kind of information, you can not only design better games, but also cross-sell existing products. One group of players can devote themselves to the clearance of the game, while another group of players can collect all the prizes before clearance, and the last family group can explore all the corners of the level before clearance. Through this combination, each player can use his favorite game method to train in the game.

Telemetry data can understand the cognitive level of players, which can change the entire gaming industry. The gaming industry has begun to use telemetry data, and it is believed that this field will make great progress in the near future. According to the effect of telemetry data analysis, the way of game production and promotion will change greatly.

Telecommunications and other industries: the value of social network data

Compared with traditional data, social network data itself is a large data source, even if in many ways, it is more like an analytical methodology. The reason for this is that the process of performing social network analysis requires dealing with already huge data sets, as well as using effective methods to increase the scale of the process by several orders of magnitude.

Some people will argue that the phone bill or text message records of all mobile phones obtained by mobile operators are big data themselves, and this kind of data can be used for a variety of purposes. However, social network analysis focuses on multiple relationship dimensions rather than a single dimension, so please climb another story. This is why social network analysis can turn traditional data sources into big data.

For the modern telephone company, it is not enough to look at the call volume, the phone company also needs to analyze the call as an independent entity. Social network analysis should first look at who is involved in the call, and then analyze it from a more in-depth perspective. We need to know not only who we called, but also who the person I called called, who they called next, and so on. To get a panoramic view of the social network, we have to reach the upper limit of what the system can handle. The navigation association between multi-tier customers and customers and multi-layer calls will double the amount of data. In addition, it also increases the difficulty of analysis, especially when using traditional tools.

The same concept applies to social networking sites. By analyzing a member of a social network, it is not difficult to analyze how related the member is, how often she sends text messages, how often she visits the site, and other indicators. However, when members have relationships with their friends, friends of friends, and friends of friends, the amount of processing required to understand the boundaries of the network is much greater.

A thousand members or users are not difficult to track. However, the direct relationship between them will rise to the million level, while considering that the "friend of a friend" will rise to the billion level. This is why social network analysis is a big data problem. Today, there are a large number of applications to analyze this relationship.

Using social network data

Social networking data and analytics have a number of far-reaching applications, one of which is changing the way companies evaluate their customers. Instead of looking only at individuals in the past, we now refer to the overall value of their network. The examples we are talking about here also apply to many other industries, where we also need to understand the relationships between people or groups, but now we are focusing on mobile phone users. because this method is the most widely used here.

Assume that the telecom operator has a relatively low-value subscriber. This user has only basic call needs and will not bring any value-added revenue to the operator. The truth is that customers who can't make a profit are worthless. In the past, the operator used to evaluate it only based on his or her personal account. In the past, if the customer called to complain or threatened to change the operator, the company might not retain him because they thought the customer was not worth retaining.

Using social network analytics, although our customer's phone bill seems to be of little value, we can identify that our customer has been on the phone with someone who is a heavyweight with a wide range of contacts. In other words, customer contact is very valuable information for operators. Research shows that once one member leaves the phone circle, other members are likely to follow, and more members begin to leave, just like an infectious disease. Soon, the members of the circle began to leave like an avalanche, which was obviously a bad thing.

Transcend personal value

One of the fascinating benefits of social networking data is that it can identify the overall income that the customer can affect, not just the direct income he or she provides. Different angles can greatly affect the decision to invest in a customer. Customers who can produce high impact need to be carefully taken care of because they can produce more value than their own direct value. If we want to maximize the overall interests of its network, the priority of this maximization should be higher than the maximization of its individual interests.

Using social network analysis, we can understand the overall value of the customer to the business in this example, not just the direct value it produces. This decision to deal with customers is completely different. The reason for telecom operators' over-investment in customers is to maintain customer networks. We can prepare business cases to maintain a wider customer base, rather than just protecting the value of individual customers.

The above example is very good, it explains how big data's analysis can generate great value in a new decision-making environment that has never existed before. Without big data, the customer would be approved to change the operator, and when his friends go with him, the telecom operator will see an avalanche of losses. Now the goal has shifted from maximizing the benefits of individual accounts to maximizing the interests of customers' social networks.

Identifying customers with a wide range of connections can also help us focus on the areas that most affect the brand image. We can give customers with a wide range of contacts the opportunity to try out freely and record their feedback. We need to make efforts to get customers to actively participate in the company's social networking sites and encourage them to write comments and express their opinions. Some companies actively recruit influential customers, giving them rewards, early trial opportunities, and other benefits. In return, influential customers continue to exert their influence, because if they are given preferential treatment, their tone tends to be more proactive.

Social networking sites such as LinkedIn or Facebook are using social network analytics to gain insight into which ads attract which users. What we care about is not only the interest expressed by the client himself, but also, equally important, what his circle of friends and colleagues are interested in. Social members will never show all their interests on social networking sites, and we can't know all the details about him. However, if a large number of friends of the customer are interested in cycling, we can infer that the customer is also interested in biking, even if he has never expressed it directly.

Law enforcement and counter-terrorism agencies can also benefit from social network analysis. We can identify people who are associated with problem groups or individuals, or even indirectly. We usually call this kind of analysis link analysis. It is possible that an individual or group, or even a club or restaurant, has something to do with the bad guys. If we find someone in multiple places with many bad people, he or she will be located, and we will think that these people deserve more in-depth monitoring and analysis. Although this will involve privacy issues, in fact this kind of analysis is already being used.

This kind of analysis is also valuable in the field of online video games. Who's playing with whom? How does the internal mode of the game change? Social network analysis expands the scope of application of telemetry data mentioned earlier. We can identify a player's preferred partner in different games. Earlier we have discussed how to classify players according to the way they play. Have those players with similar skills already formed a team to play games? Is it a mashup style that players need? If you know this kind of information, you can know whether the game manufacturer wants players to play in teams (for example, to advise players that when they log in and start playing the game, they should choose which group to join first).

There is also a lot of interesting research on how organizations communicate with each other. These studies initially focused on connections established through e-mail, phone calls and text messages. Are the departments of the company in contact with each other in the desired way? Are some employees contacting through ways other than typical channels? Who has extensive internal influence and is the best person to participate in research on how to better improve the internal communication mechanism of the company? This kind of analysis can help companies better understand the way people communicate.

The popularity and influence of social network analysis is bound to continue. Because the social network analysis process itself will maintain exponential growth, the data source will become much larger than originally thought. Perhaps the most effective function is to provide insight into the overall impact and value of the customer, and this insight can completely subvert the enterprise's view of the customer.

Summary:

Although various industries have a wide range of large data sources, they still have some common themes. Although the purpose is different, the same underlying technology is used in all industries, such as RFID.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.