Shulou(Shulou.com)06/03 Report--

Author | Yuyang Guo Yipu transferred from Quantum bit | official account QbitAI2000 multiple bug, a system riddled with holes, was used in a bank with 1.3 billion users. This is the result of a report on a major accident involving the relocation of the TSB banking system last year, from the law firm Slaughter and May. A series of Bug articles, failed tests, and the incompetence of IT service providers have led to disaster-level consequences. When the system is enabled, users find that their money is missing, spend one or two dollars to deduct thousands, and their accounts are logged in by others. The bank invested 2500 man-years in the cost of starting the system, and then spent as much as 2.8 billion to repair the pot. The system migration caused trouble for 1.3 billion users, starting with a "divorce and remarriage" case. The protagonist of divorce and remarriage is TSB, an English bank. In 2015, TSB Bank ended its 20-year marriage to Lloyds Bank, spun off from their merged group and sold itself to a new lover, the Spanish company Sabadell, for £1.7 billion, or about 15.8 billion yuan at the exchange rate at the time. However, in the past 20 years, the world has changed too much, and the banking industry has improved too much. 20 years of "marriage" has been left to TSB Bank, as well as the IT system, which is in disarray with the "ex-husband". The billions of records of TSB Bank's 5.4 million customers are still in the system of "ex-husband" Lloyds Bank, and because the fate has been broken, the system of whoring can not be wasted, and the ex-husband has to pay 100m pounds (about 930 million yuan) a year. It's as if the physical body is already with the "new lover", but the Alipay and Wechat accounts still share the same set with the "ex-husband", and have to pay the account rent for the "ex-husband", which is naturally unpleasant. So, after a long time of preparation, in 2018, they finally moved the customer information records from the "ex-husband" IT system to the new system prepared by the "new love" for TSB Bank. They set the date for the migration on the evening of Sunday, April 22nd, to first take the bank's IT system offline, and then go online after the migration, and restore customers' access to their bank accounts. They have invested more than 2500 man-years of labor costs for this migration, and CEO of Spain's New Love Group said loudly at Christmas the previous year: this is an unprecedented project in Europe, and we have invested more than 1000 professionals, which will greatly boost our growth in the UK. However, although the bosses bragged on the stage, the employees in charge of the relocation actually panicked. The relocation project was supposed to take 18 months to prepare, but as a result, the time was exceeded, the budget was also exceeded, and things were very difficult. Sure enough, Flag can't stand up too early, and the result of hitting him in the face will come soon.

At the end of the migration, the access rights of the customers were foolproof, but just 20 minutes later, they received a problem report: some customers found that their money was missing; some customers spent a little money, but their accounts were recorded as spending thousands of dollars; some customers log in and find that it is not their own account, but someone else's bank account. 1.3 billion customers had problems with their account records, so they scolded TSB Bank as a dog, and financial regulators went to the bank for tea overnight. Over the next few weeks, banks have been desperately trying to restore the system, but millions of customers have panicked and desperately withdrew their money from TSB Bank. TSB Bank, thrown into hell mode by its own mess. The root of the problem lies in testing.

Andrew Bailey, chief executive of the Financial Conduct Authority (FCA), the British financial regulator, said publicly a few weeks after the accident that a large part of the chaos in the system was due to the lack of testing, and the IBM experts hired by TSB Bank also found that TSB Bank did not adopt strict launch standards. And because the financial systems on earth are connected, the mistakes caused by the accident are permanently retained in the financial system, irreversible. The catastrophe also cost TSB Bank a lot of money. In order to compensate customers, solve the transaction of fishing in troubled waters after the system went wrong, and asked a third party for help, it cost a total of 330.2 million pounds, or about 2.84 billion yuan at the exchange rate at that time. Party B of TSB, IT provider Sabis, also received a compensation bill of 153 million pounds (more than 1.3 billion yuan) for the accident. Affected by this, TSB Bank lost 105.4 million pounds (920 million yuan) that year, and CEO Paul Pester resigned. TSB closed 86 branches in the UK at the end of November, and at least 400 jobs were lost as a result of such poor performance that banks were unsustainable. The banking system is very complex in the information age, the IT system of banks is becoming more and more complex. Sixty years ago, people could only choose to deposit cash at the counter, and ordinary customers did not have direct access to computer systems. At the time, banks used supercomputers, but they only aggregated paper data at the end of a day or week. In other words, the bank's IT system is only used by bank employees, and the interaction between banks and customers on the counter is still a paper tool. That changed in 1967. This year, the world's first automatic teller machine (ATM) was born in the UK and installed at the Barclays Enfield branch in north London. Since then, major changes have taken place in the way banks interact with customers. ITRS Group CEO Guy Warren (Guy Warren) explained: until the emergence of real ATM and online banking business, the public will not have direct access to the bank's IT system. This is just the beginning. In the era of global interconnection, the development of the Internet and mobile banking has further narrowed the gap between customers and bank IT systems, and such systems have increasingly become the key to the operation of banks. You may think that it is not difficult to log on Alipay / Wechat and show the payment code to make small money flow between banks. But in fact, behind each loading and refresh of information, there is a complex data movement-each action may be associated with many separate systems, all of which must interact with each other and connect to the core mainframe computer. The system needs to copy the data at the back end now, transfer cash from one account to another, and keep it up-to-date. And this amount of computation is multiplied by billions of times. According to the World Bank, at least 69% of the world's adults now have bank accounts. People pay bills, repay loans and subscribe to various services through bank accounts every day. Moreover, these activities are often carried out across banks, or even across borders. Multiple IT systems within a bank (mobile banking, ATM, etc.) not only need to interact with each other, but also have to establish links with banks in other countries. For example, I have a visa credit card in China, and I have to be able to spend it in the United States. The problem of migration is troublesome. TSB lies in such a high degree of complexity. In a report for TSB, IBM pointed out that the combination of new applications, the use of advanced micro-services and the use of dual-live data centers lead to compound risks in TSB production. How to correctly deal with the problems in the migration of bank IT system is a big challenge for any bank. Among them, a lot of prior planning and testing work is inevitable. Multinational banks such as HSBC have highly complex and interrelated systems that are regularly tested, migrated and updated. Even with so much experience in this area, Lancaster, a former head of IT at HSBC, admits that the trick is to get employees to spend more time on it. He also pointed out that the migration of TSB's IT system is very complicated: I'm not sure if they really realize how complicated it is. They haven't even fully figured out how to test the system. Andrew Bailey, CEO of FCA, said: TSB's accident reflects their lack of strong regression testing. Note: regression testing is a kind of software testing, which aims to check whether the original function of the software remains intact after modification, and the latest accident report has also aroused heated discussion on hacker news. Some netizens said that if TSB could choose to migrate many times on a small scale rather than a big bang one day, such a serious accident might not have happened. Take weeks / months to check during production to ensure that the structure returned by the old database is the same as that returned by the new database. Eventually, transfer all the data to the new database and shut down the old database after a period of time. The effect of doing so is better. As for the conclusion that insufficient testing led to the paralysis of the banking system, some people complained: as a test engineer, I am not surprised at all. Spending more time and putting in more people to build a better testing architecture is a "cost savings" for many companies. Managers always ask before the set launch date, "how can the test take so much time?" Something's really going to happen. They're starting to throw the pot again. Some netizens severely criticized: TSB's problem should not be said to be inadequate testing, but inadequate testing at many levels, and the lack of recoverable backup.

It has also been pointed out that the easiest way to avoid mistakes is to reduce change. The problem is that the business is evolving, both for banks and companies in other areas. From 2017 to 2018, the UK financial services sector reported an 187 per cent increase in technology disruptions, according to data released by FCA. Guy Warren believes that the system outage will not disappear. The question is, where is the acceptability? What did you think? Reference link: https://news.ycombinator.com/item?id=21870089

Https://www.tsb.co.uk/news-releases/slaughter-and-may/

Https://www.tsb.co.uk/news-releases/slaughter-and-may/slaughter-and-may-report.pdf

Https://increment.com/testing/what-broke-the-bank/

End of https://www.ft.com/content/d1f37ab2-0eb8-11ea-a225-db2f231cfeae--

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.