Shulou(Shulou.com)06/03 Report--

Has been in the industry for more than ten years, and has done a lot of distributed computing, parallel computing, memory computing, massive data processing projects, according to the current classification, these all belong to the category of cloud computing / big data. Today, let's talk about three of the projects I've done, only three. The first is the order we received for video transcoding of the video sharing website. The name of the website will not be mentioned, and it is suspected of publicizing for others. In their case, the content of video websites is played on web pages in MP4 format, but there are a variety of uploaded formats. We must uniformly convert these videos into MP4 format. We must have tried video transcoding on our own computers. Usually, a 100m video transcoding takes more than 20 minutes (CPU is Pentium IV). In order to improve the transcoding efficiency and let users get the results in close to real time, a video must be divided into multiple units immediately after receiving the video, which is dispersed to multiple computers for transcoding, and then all the transcoded videos are recovered and combined into a new video again in order. I guess video websites are all in this routine now. Our team started with streaming media at first, and the work was not difficult. We were familiar with the road and finished it quickly. As long as there are enough machines, the video can be transcoded in close to real-time. Later, it was improved again. After the video was uploaded by the user, the transcoding was carried out immediately, and after receiving a segment, the video was uploaded by the user, and our transcoding work was completed. Users can see the transcoded video they uploaded immediately, which is basically zero delay, and the stability and efficiency are higher than the requirements of the other party. After finishing this, the partner boss was very satisfied and then gave a CDN project. We still have a cooperative relationship.

The second is a bit mysterious. The project can write a paper entitled "on the trajectory and landing point of aerial moving targets". This project was sent down by XXX, which is essentially a shell attack, probably to see if it can be hit and to demonstrate the technical feasibility. At that time, people were still in a certain unit, because the research field had nothing to do with aerodynamics, so the knowledge in this area was basically illiterate. They only knew a few limited nouns and mathematical formulas, but this was just in line with the requirements of XXX, and we did not need to understand these advanced theories. They sent people to cooperate with us on this project. Our job is to take the data received by many sensors (they don't say clearly, but everyone knows that it's supposed to be radar), including the wind speed, wind direction and surface curvature of the air within the current range of kilometers. There are other indicators after collection (the amount of 3D data generated in these moments is very large), scattered to a large number of computers (100 nodes were prepared for the test) And the results must be calculated quickly (must be faster than the target moving in the air). In fact, this is very similar to the work done by SPARK and STORM today, relying on large memory, high-performance CPU, and high-speed network, skipping the bottleneck of hard disk in order to achieve fast computing. The project took a long time, mainly to communicate with partners for improvement, and then to communicate and improve again and again, over and over again. Later, it was finally completed, and it was said that the experiment had been carried out, and the feasibility was very high, and the leaders above were very satisfied with it, and they also put out a celebration wine. I didn't see it or drink it anyway. In fact, I didn't understand how they calculated it. I asked it once and said it was a secret.

The third has something to do with two barrels of oil shale gas. The current background is that a large area of natural gas has been found in Fuling, Sichuan (where mustard is also produced). According to the law of nature, oil and gas are symbiotic, so the two barrels of oil want to analyze whether there are oil resources underground in this area by collecting shale samples. if so, whether it has mining value. This program already exists, Exxon. Developed by Mobil Oil Company, it runs on CARY-1 vector machine. That set of machines is a supercomputer of the 1970s and 1980s. At that time, it could no longer meet the needs and was included in the museum. Some data analysis theories are outdated and need to be adjusted. The most unsatisfactory thing about the two barrels of oil is that the speed is too slow, and it takes hours and days to calculate the results at a time. This time it is cooperation again, two barrels of oil out of technical experts who are familiar with oil production, and our parallel computing programmers have worked together for more than half a year, which is considered to be completed. This time we use a new parallel algorithm, which is much more efficient. Like the last cannonball, the whole project started and ended quietly, and I don't know what happened later.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.