Shulou(Shulou.com)11/24 Report--

In 2018, Deepmind first released AlphaFold, a protein structure prediction database based on deep neural networks, achieving the most advanced performance in protein prediction; last year, AlphaFold 2 achieved a protein prediction rate of 98.5%; not long ago, Deepmind released a major data set update, saying that the current AlphaFold has predicted almost all known proteins.

How to effectively identify the mechanism of drug action is still a great challenge today, and the method of computational docking has been widely used to predict drug binding targets. With large-scale protein structure prediction technology, drug discovery will become easier. Therefore, since the advent of AlphaFold, there have been many voices saying that it will trigger a revolution in structural biology and revolutionize drug discovery.

In essence, AlphaFold is a tool, can we really make good use of this tool at present?

Recently, the research team from MIT gave a negative answer.

They evaluated the performance of the model for molecular docking simulation using AlphaFold2, found that the prediction ability of the model in identifying real protein-ligand interactions was weak, and proved that machine learning-based modeling was needed to improve the performance of the model in order to make better use of AlphaFold2 for drug discovery. The paper Benchmarking AlphaFold-enabled molecular docking predictions for antibiotic discovery was published in the journal Molecular Systems Biology.

Address: https://www.embopress.org/ doi / epdf / 10.15252 / msb.202211081

1 the docking calculation of so-called compounds predicted by AlphaFold 2 is a rough sequence of compounds that are most likely to bind by docking each of the candidate compounds into the target protein. This process can be completed on the set of compounds to obtain a huge virtual library, which has become a long-term goal in the field of computational chemistry.

The team first screened a group of about 39128 compounds, including known drugs (known antibiotics), active natural products and a range of other different structures. 218 compound cultures were found in the screening of Escherichia coli.

Only 218 were positive, a surprising result, but a rare number given the difficulty of finding antimicrobials.

Of the 218 active compounds, about 80% are members of known antibiotics, and the rest are a mixture of known cytotoxic compounds and some new wildcard types. This provides a good background for the experiment, because in most cases, we can predict what results we will get from reverse docking filtering.

Docking the active compounds with the predicted protein structure, the team studied the potential binding targets of these active compounds. Over the years, consensus scores for a large number of genome knockout scans in E. coli have yielded 296 basic proteins, so it is reasonable to infer that all the target proteins that really inhibit growth may be on these lists.

The authors docked 218 active compounds with 296 basic Escherichia coli protein structures predicted by AlphaFold 2, and calculated the combinations of 218 compounds with 296 proteins by several different calculation methods. The binding posture and binding affinity of more than 64000 protein-ligand pairs were predicted.

▲ diagram note: molecular docking on AlphaFold structure

From the point of view of the amount of calculation, this is a test with high reliability, especially considering the number of internal controls (compounds with known targets and compounds with known binding conformations in these targets), this calculation is very valuable.

As a control, the team also randomly selected 100 compounds that had no inhibitory effect on bacterial growth from a group of compounds for the same calculation, thus obtaining the prediction of the binding posture and affinity of 29600 protein-ligand pairs.

2 the performance of the model based on the predicted structure of AlphaFold 2 is very weak, although this work predicts the hybridity of compounds with proteins, including active and inactive compounds, the question is, how many of these predictions are false positive?

Compare model predictions with known antibiotic binding targets in order to evaluate the performance of the model methods used, the authors compare model predictions with known interactions of commonly used antibiotic classes.

The authors collected antibiotic-protein target pairs in the previous literature to form a data set containing 142 kinds of antibiotic-protein interactions. The results showed that their model correctly predicted only three interactions with strong binding (the binding affinity threshold was-7 kcal / mol) and 43 interactions with general binding (the binding affinity threshold was-5 kcal / mol). Therefore, the true positive rates predicted by the model were 2.1% and 30.3%, respectively.

This comparison shows that the performance of the modeling platform based on AlphaFold 2 prediction structure is very weak.

To measure the enzyme inhibition of 12 basic proteins, the author then selected 12 basic proteins, which can be used for enzymatic determination. By measuring the enzyme inhibition of 218 active compounds on these proteins, the authors further evaluated the subset predicted by the model.

Note: the average relative activity of all 218 active compounds and 12 proteins have been tested by inhibition test. Binding interaction hits are protein-ligand interactions (red dots), and all other interactions are designated as non-hits (gray dots).

The results showed that the basic proteins in all tests were inhibited by at least four different compounds, covering a series of binding affinity thresholds from strong to weak, and the docking predicted by the model based on AlphaFold 2 showed a wide range of hybridity.

Finally, the author makes a statistical benchmark test on the performance of the modeling platform. Based on the measured data of enzyme inhibition, the binding affinity thresholds were set to less than-5 kcal / mol and-7 kcal / mol by comparing the observed binding hits with their predicted interactions.

The results showed that the stronger affinity threshold led to less prediction of binding interaction, and the true positive rate was lower and the accuracy was higher. The performance of the model varies from weak to medium according to the set binding affinity threshold.

Under the condition of independent of the binding affinity threshold, the receiver operating characteristic curve (ROC) and the exact recall curve (PR) were used to evaluate again, which also showed that the performance of the model was weak.

In short, there are not only a large number of false positives in the prediction results of the model (that is, inactive compounds are predicted to bind to the active sites of key bacterial proteins), but also a large number of false negatives (that is, interactions are known to exist, but not found). Only when the most stringent binding affinity threshold is reached, the model will perform slightly better than the random prediction.

3 AlphaFold itself is right. Making good use of machine learning methods is the key. The next question is, is the weak performance of the model caused by the quality of the protein structure provided by AlphaFold2?

The problem comes from the docking method rather than the protein structure quality. in order to verify this problem, the authors repeated docking simulations of 218 active compounds with each of the eight protein structures determined by the experiment, and also benchmark the performance of the model. The results show that the auROC values are similar in quantity, ranging from 0.25 (glmU) to 0.69 (gyrAB), with an average of 0.46. Similar results were found for auPRC values, ranging from 0.03 (ligA) to 0.56 (gyrAB), with an average of 0.22.

These findings show that the molecular docking of the predicted structure using AlphaFold2 is similar to the structure determined by experiments. This is also consistent with the previous fidelity evaluation of the protein structure determined by AlphaFold, which shows that the weak performance of the model is due to the docking method, not the poor quality of the protein structure.

The use of machine learning methods can improve the performance of the model based on the weak performance of molecular docking, and the research team explored ways to improve performance.

In this study, four different scoring functions based on machine learning, namely RF-Score, RF-Score-VS, PLEC score and NNScore, are used to benchmark and improve the performance of the model.

Compared with the virtual screening adaptability of RF-Score and RF-Score-VS-RF-Score, which uses the combination of random forests or decision trees to predict the binding affinity between proteins and ligands, PLEC score uses extended connection fingerprints between protein-ligand pairs, and NNScore is a set based on neural networks.

In the study, the authors used a scoring function to train the enhanced (DUD-E) database using PDBbind v2016 or a useful bait catalog to re-evaluate the docking posture predicted by AutoDock Vina.

In addition, the study used DOCK6.9 and each machine learning-based score function applied to the AutoDock Vina posture, predicted the binding affinity between each antibacterial compound and each of the 12 essential proteins for empirical testing, and benchmarked the performance of each method. The test results show that the average auROC value is between 0.46 and 0.63 (figure A below).

Among them, docking with DOCK6.9 and using PLEC score to re-score the AutoDock Vina pose averagely, the auROC value is lower than that of using AutoDock Vina alone, and the auROC value of DOCK6.9 is 0.46 (range 0.25 to 0.61) and 0.47 (range PLEC score 0.28 to 0.63) (figure A below)

In contrast, re-grading AutoDock Vina poses with RF-Score, RF-Score-VS, or NNScore can improve model performance, with average auROC values of 0.62 (range 0.53 to 0.69), 0.63 (range 0.46 to 0.75) and 0.58 (range 0.41 to 0.69), respectively. The results are similar to those of auPRC, with an average of 0.24 when re-scored with RF-Score.

The performance evaluation of these models shows that some scoring functions based on machine learning improve the prediction accuracy.

▲ Note: use machine learning to benchmark and improve the performance of the model. a. In different molecular docking programs and different machine learning-based posture scoring functions. The white dot represents the average; the range of the 25th-75th percentile of the grey bar table; the grey box line represents the range of values that are not regarded as outliers; and the horizontal line at 0.5 represents the benchmark generated by the random forecast. b. The sorted binding affinity of protein-ligand pairs is modeled by using a machine learning-based re-scoring function in AutoDock Vina. The curve is colored according to the re-scoring function used in (A), and the shaded area represents the binding affinity threshold of > 7. Cmure E. The dependence of predictive accuracy, predictive positive number (protein-ligand interaction) and true positive rate / false positive rate on the number of models used.

The group intelligence method can improve the prediction accuracy because some scoring functions based on machine learning will increase auROC and auPRC. The study also discussed whether the prediction accuracy and true positive rate can be improved by using the re-scoring model combined with the "group intelligence" method under the condition that the binding affinity threshold is strictly limited.

The predicted protein-ligand interaction is defined as satisfying the binding affinity threshold of all models, and the AutoDock Vina prediction is combined with the prediction of the above four scoring functions based on machine learning. By using this consensus method, the study found that the prediction accuracy could be improved with the number of models used (figure C above), which was consistent with the expected corresponding reduction in the predicted number of protein-ligand interactions (figure D above).

At the same time, the ratio of true positive rate to false positive rate increased with the increase in the number of models used, which was unexpected (figure E above).

It can be seen that this result is consistent with the discovery that some scoring functions based on machine learning are used to improve the predictive ability, which further indicates that the combination of molecular docking and machine learning-based models can make better use of the protein structures predicted by AlphaFold2 for drug screening.

Therefore, some machine learning methods can indeed improve the accuracy of prediction. However, this is only a partial success, and the data set used in the current study contains a lot of experimental facts about proteins and compounds that have been identified, and it is not known whether these methods will still work if they are involved in areas of less concern.

Although AlphaFold provides us with a large and reasonable protein structure, our ability to realize its value is still very limited. So at least for now, the claim that "AlphaFold will revolutionize drug discovery" remains to be confirmed, and success lies in the future.

Reference link:

Https://www.science.org/content/blog-post/not-alphafold-s-fault

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.