Shulou(Shulou.com)06/03 Report--

Memory calculation refers to the calculation method in which the data is stored in memory in advance and the results in the middle of each step do not fall on the hard disk, which is suitable for the situation with high performance requirements and large concurrency.

HANA, TimesTen and other in-memory databases can realize in-memory computing, but these products are expensive, complex structure, difficult to implement, and high total cost of ownership. The aggregator introduced in this paper can also realize memory calculation, and its structure is simple and convenient to implement, so it is a lightweight memory computing engine.

Let's introduce the general process of implementing memory calculation by the aggregator.

First, start the server

Aggregators can be deployed in two ways: stand-alone deployment and embedded deployment. The first difference is that the startup mode is different.

L standalone deployment

When deploying as a stand-alone service, the aggregator and the application system use different JVM, and they can be deployed on the same machine or separately. Application systems usually use aggregator drivers (ODBC or JDBC) to access collective computing services, or they can be accessed through HTTP.

Start the stand-alone service under n Windows, execute "install directory\ esProc\ bin\ esprocs.exe", and then click the "start" button.

The installation directory / esProc/bin/ServerConsole.sh should be executed under n Linux.

For details of starting the server and configuration parameters, please refer to: http://doc.raqsoft.com.cn/esproc/tutorial/fuwuqi.html.

L embedded deployment

When deployed as an embedded service, the aggregator can only integrate with JAVA applications, and the two share JVM. The application system accesses the embedded centralized computing service through JDBC and does not need to start it specially.

For details, please see http://doc.raqsoft.com.cn/esproc/tutorial/bjavady.html.

Second, load data

Loading data refers to the process of reading external data such as database, log, WebService and so on into memory through aggregator script.

For example, the order table in Oracle is as follows:

Order ID (key)

Customer ID

Order date

Freight charge

10248

VINET

2012-07-04

32.38

10249

TOMSP

2012-07-05

11.61

10250

HANAR

2012-07-08

65.83

10251

VICTE

2012-07-08

41.34

10252

SUPRD

2012-07-09

51.3

...

...

...

...

The details of the order are as follows:

Order ID (key) (fk)

Product ID (key)

Unit price

Quantity

10248

seventeen

fourteen

twelve

10248

forty-two

nine

ten

10248

seventy-two

thirty-four

five

10249

fourteen

eighteen

nine

10249

fifty-one

forty-two

forty

...

...

...

...

To load the above two tables into memory, you can use the following aggregate script (initData.dfx):

A

one

= connect ("orcl")

two

= A1.query ("select order ID, customer ID, order date, freight from order") .keys (order ID)

three

= A1.query@x ("select order ID, product ID, unit price, quantity from order details") .keys (order ID, product ID)

four

= env (order, A2)

five

= env (order details, A3)

A1: connect to the Oracle database.

A2-A3: execute the SQL query to take out the order table and the order schedule, respectively. Query@x indicates that the connection is closed after SQL is executed. The function keys establishes a primary key, which is not required if the database already has a primary key defined.

A4-A5: place the two tables in memory and name them order details and order details for future reference in business calculations. The function env is to set / release global shared variables so that they can be referenced by other algorithms under the same JVM. Here, the memory table is set as a global variable, that is, the whole table data is stored in memory for use by other algorithms. In fact, resources such as external tables and file handles can also be set as global variables so that the variables reside in memory.

The script needs to be executed to take effect.

For embedded centralized services, scripts are usually executed when the application system starts. If the application system is a JAVA program, you can execute initData.dfx through JDBC in the program. The key code is as follows:

1. Com.esproc.jdbc.InternalConnection con=null

2. Try {

3. Class.forName ("com.esproc.jdbc.InternalDriver")

4. Con = (com.esproc.jdbc.InternalConnection) DriverManager.getConnection ("jdbc:esproc:local://")

5. ResultSet rs = con.executeQuery ("call initData ()")

6.} catch (SQLException e) {

7. Out.println (e)

8.} finally {

9. If (contextual null) con.close ()

10.}

This article describes in detail the process of JAVA calling the aggregator, http://doc.raqsoft.com.cn/esproc/tutorial/bjavady.html

If the application system is JAVA WebServer, you need to write a Servlet, execute initData.dfx through JDBC in the init method of Servlet, set the servlet as the startup class, and configure it in web.xml as follows:

MyServlet

Com.myCom.myProject.myServlet

three

For independently deployed centralized servers, JAVA applications also use the JDBC interface to execute aggregator scripts, similar to embedded services. The difference is that the script is stored remotely, so you need to specify the server address and port as follows:

St = con.createStatement ()

St.executeQuery ("= callx (\" initData.dfx\ "; [\" 127.0.0.1 initData.dfx 8281\ "])")

If the application system is not a JAVA architecture, you should use ODBC to execute the aggregator script, as detailed in http://doc.raqsoft.com.cn/esproc/tutorial/odbcbushu.html

For independently deployed servers, you can also manually execute initData.dfx on the command line, away from the application. In this case, you need to write another script (such as runOnServer.dfx):

A

one

= callx ("initData.dfx"; ["127.0.0.1 purl 8281"])

Then call runOnServer.dfx with esprocx.exe on the command line:

D:\ raqsoft64\ esProc\ bin > esprocx runOnServer.dfx

The usage under Linux is similar. Refer to http://doc.raqsoft.com.cn/esproc/tutorial/minglinghang.html.

Third, perform the operation to obtain the result

After the data is loaded into memory, you can write various algorithms to access, perform calculations and obtain results. Here is an example: take the customer ID as a parameter to count the customer's monthly orders each year.

The SQL statement in the Oracle corresponding to the algorithm is as follows:

Select to_char (order date, 'yyyy') AS year, to_char (order date,' MM') AS month, count (1) AS order quantity

From order

Where customer ID=?

Group by to_char (order date, 'yyyy'), to_char (order date,' MM')

In the aggregator, you should write the following business algorithm (algorithm_1.dfx)

A

one

= order .select @ m (customer ID==pCustID) .groups (year (order date): year, month (order date): month; count (1): order quantity)

For ease of debugging and maintenance, you can also write step by step:

A

one

= order .select @ m (customer ID==pCustID)

two

= A1.groups (year): year, month (order date): month

Count (1): order quantity)

A1: filter data by customer ID. Where "order" is the global variable defined when loading data, pCustID is an external parameter, which is used to specify the customer ID to be counted, and the function select executes the query. @ m means parallel computing, which can significantly improve performance.

A2: perform a group summary and output the calculation results. By default, the aggregator returns the last cell with the expression, which is A2. If you want to return the value of the specified unit, you can use the return statement

When pCustID= "VINET", the calculation results are as follows:

Year

Month

Order quantity

2012

seven

three

2012

eight

two

2012

nine

one

2013

eleven

four

It should be noted that if multiple business calculations have to query the customer ID, you might as well sort the order by the customer ID when loading the data, so that dichotomy can be used in the subsequent business algorithm for fast query, that is, the select@b function. In terms of implementation, SQL in initData.dfx should be changed to:

= A1.query ("select order ID, customer ID, order date, freight from order order by customer ID")

Accordingly, the query in algorithm_1.dfx should be changed to:

= order .select @ b (customer ID==pCustID)

The method of executing scripts to get results, as mentioned earlier, focuses on reports, the most commonly used applications.

Because report tools have visual design interfaces, there is no need to use JAVA code to call the aggregator, just configure the data source to point to the aggregator service, and call the aggregator script in the form of stored procedures in the report tool.

For embedded centralized servers, the invocation statement is as follows:

Call algorithm_1 ("VINET")

Because the algorithm in this example is very simple, you can actually write the expression in SQL directly in the report without writing a separate dfx script:

= order .select @ m (customer ID== "VINET") .groups (year (order date): year, month (order date): month; count (1): order quantity)

For independently deployed aggregate servers, the remote invocation statement is as follows:

= callx ("algorithm_1.dfx", "VINET"; ["127.0.0.1 virtual 8281"])

Sometimes, there are few business algorithms that need to be carried out in memory, and it is not convenient for web.xml to add startup classes, so you can call the initialization script in the business algorithm to achieve the effect of automatic initialization and save the process of writing servlet. The specific script is as follows:

A

B

one

If! ifv (order)

= call ("initData.dfx")

two

= order .select @ m (customer ID==pCustID)

three

= A2.groups (year): year, month (order date): month

Count (1): order quantity)

A1-B1: determines whether the global variable "order details" exists, and if not, executes the initialization data script initData.dfx.

A2-A3: continue with the original algorithm.

IV. Quoting thinking

The previous example uses the select function, which works similar to SQL's where statement and allows conditional queries, but the underlying principles of the two are quite different. The where statement copies the data each time to generate a new result set, while the select function simply refers to the original record pointer and does not copy the data. Take querying an order by customer as an example, the difference between quoting and copying is shown in the following figure:

As you can see, due to the use of the reference mechanism, the aggregator takes up less space and has higher computing performance (faster memory allocation). In addition, the above calculation results can be queried again, the new result set in the aggregator also refers to the original records, and SQL will copy a lot of new records.

In addition to queries, there are many aggregator algorithms that use reference thinking, such as sorting, set intersection and complement, association, merging.

Fifth, common calculation

Reviewing the previous case, you can see that there is a corresponding relationship between the aggregator statement and the SQL statement as follows:

Calculate

SQL

Aggregator

Query

Select

Select

Conditions

Where... . Order. Customer ID=?

Order ID. Customer ID==pCustID

Group summary

Group by

Groups

Date function

To_char (order date, 'yyyy')

Year (order date)

Alias

AS year

Year

In fact, the aggregator supports sophisticated structured data algorithms, such as:

L GROUP BY... HAVING

A

one

= order .groups (year (order date): year; count (1): order quantity) .select (order quantity > 300)

L ORDER BY... ASC/DESC

A

one

= order .sort (customer ID,- order date)

/ sorting only changes the order of the record pointer and does not copy the record

L DISTINCT

A

one

= order .id (year (order date))

/ take a unique value

two

= A1. (customer ID)

/ all occurrence values

three

= order. ([year (order date), customer ID])

/ all occurrence values of the combination

L UNION/UNION ALL/INTERSECT/MINUS

A

one

= order .select (freight > 100)

two

= order .select ([2011jue 2012] .pos (year (order date))

three

= A2 | A3

/ UNION ALL

four

= A2&A3

/ UNION

five

= A2 ^ A3

/ INTERSECTION

six

= A2\ A3

/ DIFFERENCE

Unlike the merge complement of SQL, the aggregator only combines record pointers and does not copy records.

L SELECT... FROM (SELECT …)

A

one

= order .select (order date > date ("2010-01-01"))

/ execute query

two

= A1.count ()

Re-statistics of a pair of result sets

L SELECT (SELECT... FROM) FROM

A

one

= order .new (order ID, customer .select (customer ID== order. Customer ID). Customer name)

/ customer table and order table are global variables

L CURSOR/FETCH

Cursors can be used in two ways. One is that an external JAVA program calls the aggregator, which returns the cursor, such as the following script:

A

one

= order .select (order date > = date ("2010-01-01") .select ()

JAVA can continue processing after obtaining the cursor, in the same way that JDBC accesses the cursor.

Second, use cursors inside the aggregator to traverse and complete the calculation. For example, the following script:

A

B

one

= order .cursor ()

two

For A1100

= A2.select (order date > = date ("2010-01-01"))

/ take 100 operations at a time

three

...

The aggregator is suitable for solving the calculation of complex business logic, but considering that simple algorithms are in the majority and many programmers are used to using SQL statements, the aggregator also supports the so-called "simple SQL" syntax. For example, algorithm_1.dfx can also write:

A

one

$() select year (order date) AS year, month (order date) AS month, count (1) AS order quantity

From {order}

Where order. Customer ID='VINET'

Group by year (order date), month (order date)

The above script is common to any SQL,$ () that executes the SQL statement of the default data source (aggregator). If you specify a data source name such as $(orcl), you can execute the SQL statement of the corresponding database (the Oracle database whose data source name is orcl).

The from {} statement can take a number from any aggregator expression, such as: from {order .groups (year (order date): year; count (1): order quantity)}

From can also be fetched from files or excel, such as from d:/emp.xlsx

Simple SQL also supports join. On... Statement, but because the SQL statement (which refers to any RDB) performs poorly in the association algorithm, it is not recommended to use it easily. For association operations, the aggregator has a special high-performance implementation method, which will be described in the following chapters.

For more information on simple SQL, please see http://doc.raqsoft.com.cn/esproc/func/dbquerysql.html#db_sql_.

VI. Orderly citation

SQL does operations based on unordered sets, so it can not take numbers directly, but can only generate sequence numbers temporarily, which is inefficient and tedious to use. Different from the SQL system, aggregators can operate based on ordered sets and can directly take numbers with ordinal numbers. For example:

A

one

= order .sort (order date)

/ if it is sorted when loading, this step can be omitted

two

= A1.m (1). Order ID

/ Article 1

three

= A1.m (- 1). Order ID

/ Last item

four

= A1.m (to (3pm 5))

/ articles 3-5

The function m () can get the member by the specified sequence number, and the parameter is negative to indicate the reverse order. The parameter can also be a collection, such as m ([3pr 4pr 5]). By using the function to (), the set can be generated according to the starting and ending sequence number, to (3Magol 5) = [3Mague 4Mague 5].

The previous example mentioned the dichotomy query select@b, which actually takes advantage of the orderly access of the aggregator.

Sometimes we want to take the top N, the general idea is to sort first, and then take the top N members by location. The aggregator script is as follows:

= order .sort (order date) .m (to)

The corresponding SQL is written as follows:

Select top * from order order by order date-MSSQL

Select * from (select * from order order by order date) where rownum=1000, freight * 0.8, freight)

VII. Correlation calculation

Association computing is the core algorithm of relational database, which is widely used in in-memory computing, such as statistics of annual and monthly order quantity and order amount.

The corresponding SQL statement of Oracle is:

Select to_char (order. Order date, 'yyyy') AS year, to_char (order. Order date, 'MM') AS month, sum (order details. Unit price * order details. Quantity) AS sales amount, count (1) AS order quantity

From order details left join order on order details. Order ID= order. Order ID

Group by to_char (order. Order date, 'yyyy'), to_char (order. Order date, 'MM')

When the above algorithm is implemented with the aggregator, the script for loading the data remains the same, and the business algorithm is as follows (algorithm_2.dfx)

A

one

= join (order details: child table, order ID; order: main table, order ID)

two

= A1.groups (year (main table. Order date): year, month (main table. Order date): month; sum (child table. Unit price * subtable. Quantity): sales amount, count (1): order quantity)

A1: associate the order details with the order. The main table of the child table is an alias. Click on the cell to see the result as follows

As you can see, the join function of the aggregator works the same as the SQL join statement, but the structural principle is quite different. The field value of the result formed by the association of the function join is not an atomic data type, but a record. Number to express relational references, multi-layer association is very convenient.

A2: group summary.

The calculated results are as follows:

Year

Month

Sales amount

Order quantity

2012

seven

28988

fifty-seven

2012

eight

26799

seventy-one

2012

nine

27201

fifty-seven

2012

ten

37793.7

sixty-nine

2012

eleven

49704

sixty-six

...

...

...

...

The association relationship is divided into many categories, and the above order and order details belong to one of them: master-child association. For master-child association, as long as they are sorted by the associated fields when loading data, the merging algorithm can be used to improve the performance in the business algorithm. For example:

= join@m (order details: child table, order ID; order: main table, order ID)

The function join@m represents a merge association and is valid only for two or more tables in the same order.

The association calculation of the aggregator is different from that of RDB. RDR uses the same algorithm for all types of association relations, so it is unable to carry out targeted optimization. While the aggregator adopts the concept of divide and conquer, it provides different algorithms for different types of association relations, which can be optimized transparently.

In addition to master-child association, the most commonly used is foreign key association, and the commonly used foreign key tables (or dictionary tables) are classification, region, city, employee, customer, and so on. For the foreign key association, the aggregator also has the corresponding optimization method, that is, to establish the association in advance in the data loading phase, so that the business algorithm does not have to be temporarily associated, so the performance is improved, especially in the case of concurrency. In addition, the aggregator uses pointers to establish foreign key associations, resulting in faster access.

For example, in this case: the customer ID field of the order table is a foreign key. Corresponding to the customer table (customer ID, customer name, region, city), you need to calculate the number of orders in each region and city.

Data loading script (initData_3.dfx) is as follows:

A

one

= connect ("orcl")

two

= A1.query ("select order ID, customer ID, order date, freight from order") .keys (order ID)

three

= A1.query@x ("select customer ID, regional, city from customer") .keys (customer ID)

four

= A2.switch (customer ID,A3: customer ID)

five

= env (order, A2)

six

= env (customer, A3)

A4: use the function switch to establish a foreign key association and replace the customer ID field of the order table with the pointer of the corresponding record of the customer table.

The business algorithm script is as follows (algorithm_3.dfx)

A

one

= order .groups (customer ID. Region: region, customer ID. City: city; count (1): order quantity)

A foreign key pointer association has been established when the data is loaded, so the "customer ID" in A1 indicates that the customer table record that the customer ID field of the order table points to, and "customer ID. Region" is the region field of the customer table.

"." is used many times in the script. The syntax is more intuitive and easier to understand than SQL, especially when it comes to multi-table and multi-layer association. In SQL, more than one connection is like the Book of Heaven.

The above calculation results are as follows:

Area

City

Order quantity

Northeast China

Dalian

forty

East China

Nanjing

eighty-nine

East China

Nanchang

fifteen

East China

Changzhou

thirty-five

East China

Wenzhou

eighteen

...

...

...

VIII. Internal and external mixed calculation

Although memory computing is fast, memory is limited, so it usually resides only the data that is most commonly used and accessed concurrently, while the data that is not stored or accessed with low frequency should be left on the hard disk and temporarily loaded when used. And participate in the calculation together with the in-memory data. This is the so-called internal and external hybrid computing.

The following example illustrates the internal and external hybrid calculation in the aggregator.

Case description: in a retail industry system, the access frequency of order details is low, and the amount of data is large, so it is unnecessary and impossible to reside in memory. Now associate the order details with the orders in memory and count the annual sales of each product.

Data loading script (initData_4.dfx) is as follows:

A

one

= connect ("orcl")

two

= A1.query@x ("select order ID, customer ID, order date, freight from order order by order ID") .keys (order ID)

four

= env (order, A2)

The business algorithm script (algorithm_4.dfx) is as follows:

A

one

= connect ("orcl")

two

= A1.cursor@x ("select order ID, product ID, quantity from order detail order by order ID")

three

= order .cursor ()

four

= joinx (A2: child table, order ID; A3: main table, order ID)

five

= A4.groups (year (main table. Order date): year, sub-table. Product ID: product; sum (subtable. Quantity): sales quantity)

A2: execute the SQL to get the order details with a cursor in order to calculate a large amount of data that far exceeds the memory.

A3: convert the order table to cursor mode, which will be used in the next step.

A4: associate order schedule and order table. The function joinx is similar to join@m in that it can merge and associate ordered data, except that the former is valid for cursors and the latter is valid for ordinal tables.

A5: perform group summarization.

IX. Data update

The physical table in the database always changes, and this change should be reflected in the shared memory table in time to ensure that the memory calculation results are correct. In this case, the memory needs to be updated. If the physical table is small, it is easy to solve, as long as the initialization data script (initData.dfx) is executed regularly. However, this cannot be done if the physical table is too large, because the initialization script is fully loaded, itself consumes a lot of time, and memory calculations cannot be performed at load time. For example, a retail giant has a large amount of order data, which is usually loaded from the database to memory for more than 5 minutes, but in order to ensure real-time performance, the memory data needs to be updated every 5 minutes. Obviously, there is an obvious contradiction between the two.

The solution is natural: when the physical table is too large, it should be updated incrementally. The 5-minute incremental business data is usually very small, and the increment will not affect the efficiency of updating memory.

To implement incremental updates, you need to know what incremental data is, no more than the following three methods:

Method A: add a tag field to the original table to identify. The disadvantage is that the original table will be changed.

Method B: create a "change table" in the original database and record the changed data. The advantage is not to move the original table, but the disadvantage is that it still has to move the database.

Method C: record the change table in another database, or in the text file Excel. The advantage is that no changes are made to the original database, but the disadvantage is that it increases the maintenance workload.

The aggregator supports multi-data source computing, so there is no essential difference between method B and C, so let's take B as an example to update the order table.

The first step is to establish an "order change table" in the database, inherit the original table field, and add a new "change tag" field. When the user modifies the original table, the record needs to be synchronized in the change table. The order change table shown below indicates that 1 item is added, 2 items are modified and 1 item is deleted.

Order ID (key)

Customer ID

Order date

Freight charge

Change mark

10247

VICTE

2012-07-08

one hundred and one

Add

10248

VINET

2012-07-04

one hundred and two

Modify

10249

TOMSP

2012-07-05

one hundred and three

Modify

10250

HANAR

2012-07-08

one hundred and four

Delete

The second step is to write the aggregator script updatemem_4.dfx to update the data.

A

B

one

= connect ("orcl")

two

= order cp= order .derive ()

three

= A1.query ("select order ID, customer ID, order date, shipping fee, change mark from order change")

four

= order deletion = A3.select (change mark = "delete")

= order cp.select (order deleted. (order ID) .contain (order ID))

five

= order cp.delete (B4)

six

= order add = A3.select (change tag = = "add")

= order cp.insert@f (0: order added)

seven

= order modification = A3.select (change tag = = "modify")

= order cp.select (order modification. (order ID) .pos (order ID))

eight

= order cp.delete (B7)

nine

= order cp.insert@f (0: order modification)

ten

= env (order, order cp)

eleven

= A1.execute ("delete from order change")

twelve

= A1.close ()

A1: establish a database connection.

A2: make a copy of the order in memory and name it order cp. The following procedure only modifies the order cp, and then replaces the order in memory. During this period, the order can still be calculated normally.

A3: take the database order change table.

A4-B5: take out the records that need to be deleted in the order change table, find them in the order cp, and delete them.

A6-B6: take out the new records to be added in the order change table and add them to the order cp.

A7-B9: this step is to modify the order cp, which is equivalent to deleting it first and then appending it. You can also use the modify function to implement the modification.

A10: the modified order cp will be resident in memory and named as the order.

A11-A12: clear the change Table so that you can fetch a new change record next time.

The above script implements a complete data update, but in many cases you only need to append the data, which makes the script much easier.

After the script is written, you need the third step: schedule the script to be executed for 5 minutes.

There are many ways of timing execution. If the aggregator is deployed as a stand-alone service and does not share JVM with Web applications, you can use the timing tools (scheduled tasks or crontab) that come with the operating system to periodically execute aggregator commands (esprocx.exe or esprocx.sh).

Some web applications have their own timed task management tools that can regularly execute a JAVA class, so you can write a JAVA class and call the aggregator script with JDBC.

If the web application does not have a timing task management tool, it needs to implement the timing task manually, that is, write the JAVA class, inherit the built-in timing class TimerTask of java, call the aggregator script in it, and then invoke the timing task class in the startup class.

Where the startup class myServle4 is:

1. Import java.io.IOException

2. Import java.util.Timer

3. Import javax.servlet.RequestDispatcher

4. Import javax.servlet.ServletContext

5. Import javax.servlet.ServletException

6. Import javax.servlet.http.HttpServlet

7. Import javax.servlet.http.HttpServletRequest

8. Import javax.servlet.http.HttpServletResponse

9. Import org.apache.commons.lang.StringUtils

10. Public class myServlet4 extends HttpServlet {

11. Private static final long serialVersionUID = 1L

12. Private Timer timer1 = null

13. Private Task task1

14. Public ConvergeDataServlet () {

15. Super ()

16.}

17. Public void destroy () {

18. Super.destroy ()

19. If (timer1 invalid null) {

20. Timer1.cancel ()

21. }

twenty-two。 }

23. Public void doGet (HttpServletRequest request, HttpServletResponse response)

24. Throws ServletException, IOException {

25. }

twenty-six。 Public void doPost (HttpServletRequest request, HttpServletResponse response)

twenty-seven。 Throws ServletException, IOException {

twenty-eight。 DoGet (request, response)

twenty-nine。 }

thirty。 Public void init () throws ServletException {

thirty-one。 ServletContext context = getServletContext ()

thirty-two。 / / scheduled refresh time (5 minutes)

thirty-three。 Long delay = new Long (5)

thirty-four。 / / start the timer

thirty-five。 Timer1 = new Timer (true)

thirty-six。 Task1 = new Task (context)

thirty-seven。 Timer1.schedule (task1, delay * 60 * 1000, delay * 60 * 1000)

thirty-eight。 }

thirty-nine。 }

The scheduled task class Task is:

11. Import java.util.TimerTask

12. Import javax.servlet.ServletContext

13. Import java.sql.*

14. Import com.esproc.jdbc.*

15. Public class Task extends TimerTask {

16. Private ServletContext context

17. Private static boolean isRunning = true

18. Public Task (ServletContext context) {

19. This.context = context

20.}

21. @ Override

twenty-two。 Public void run () {

23. If (! isRunning) {

24. Com.esproc.jdbc.InternalConnection con=null

25. Try {

twenty-six。 Class.forName ("com.esproc.jdbc.InternalDriver")

twenty-seven。 Con = (com.esproc.jdbc.InternalConnection) DriverManager.getConnection ("jdbc:esproc:local://")

twenty-eight。 ResultSet rs = con.executeQuery ("call updatemem_4 ()")

twenty-nine。 }

thirty。 Catch (SQLException e) {

thirty-one。 Out.println (e)

thirty-two。 } finally {

thirty-three。 / / close the dataset

thirty-four。 If (contextual null) con.close ()

thirty-five。 }

thirty-six。 }

thirty-seven。 }

thirty-eight。 }

10. Comprehensive examples

Next, through a comprehensive example, let's take a look at how the aggregator implements memory computing well in the case of multiple data sources and complex algorithms:

Case description: a B2C website needs to try to calculate the total mailing cost of the order in order to select the appropriate postage rules at a certain cost. In most cases, the postage is determined by the total weight of the package, but when the price of the order exceeds the specified value (such as $300), free shipping is provided. As a result, you need to output the mailing cost of each order as well as the total cost.

The order table has been loaded into memory, as follows:

Id

Cost

Weight

Josh2

one hundred and fifty

six

Drake

one hundred

three

Megan

one hundred

one

Josh3

two hundred

three

Josh4

five hundred

one

The postage rules are different in each trial calculation, so the parameter "pRule" is temporarily passed in in the format of json string. One time the rules are as follows:

[{"field": "cost", "minVal": 300, "maxVal": 1000000, "Charge": 0}

{"field": "weight", "minVal": 0, "maxVal": 1, "Charge": 10}

{"field": "weight", "minVal": 1, "maxVal": 5, "Charge": 20}

{"field": "weight", "minVal": 5, "maxVal": 10, "Charge": 25}

{"field": "weight", "minVal": 10, "maxVal": 1000000, "Charge": 40}]

The above json string indicates the postage of each field when it is in a range of values. The first record indicates that when the cost field is between 300 and 1000000, the postage is 0 (free shipping); the second record says that when the weight field is between 0 and 1 (kg), the postage is 10 (US dollars).

Idea: convert the json string into a two-dimensional table, find out the records whose filed fields are cost and weight respectively, and then cycle through the entire order table. In the cycle, we first judge whether the cost value in the order record meets the free standard, and if not, the postage grade is judged according to the weight, and then the postage is calculated. After calculating the postage for each order, the total postage is calculated, and the summary result is appended to the last record in the order table.

The data loading process is simple, and I won't repeat it here, that is, read the database table and name it "order table".

Business algorithms are relatively complex, as follows:

A

B

C

D

one

= pRule.export@j ()

/ parse json and turn to two-dimensional table

two

= Free = A1.select (field== "cost")

/ take the free standard, single

three

= charge = A1.select (field== "weight") .sort (- minVal)

/ pick up the charging ladder, multiple

four

= order form .derive (postage)

/ copy and add fields

five

For A4

If is free. MinVal

< A5.cost >

A5. Postage= is free. Charge

six

Next

seven

For charge

If A5.weight > B7.minVal

> A5.postage=B7.Charge

eight

Next A5

nine

= A4.record (["sum", A4.sum (postage)])

A1: parse json and turn it into a two-dimensional table. The aggregator supports multiple data sources, not only RDB, but also NOSQL, file, webService.

A2-A3: inquire about the postage rules, which are divided into two types: free and charge.

A4: add an empty field postage.

A5-D8: cycle through the order form according to two rules, calculate the corresponding postage, and fill in the postage field. Process control is used here, and the aggregator is represented by indentation, in which A5 and B7 are loop statements, C6 and D8 jump into the next loop, and B5 and C7 are judgment statements.

A9: add a new record to the order form and fill in the total value.

The calculated results are as follows:

Id

Cost

Weight

Postage

Josh2

one hundred and fifty

six

twenty-five

Drake

one hundred

three

twenty

Megan

one hundred

one

ten

Josh3

two hundred

three

twenty

Josh4

five hundred

one

0

Sum

seventy-five

So far, this paper introduces in detail the complete process of using the aggregator as a memory computing engine, including common calculation methods and advanced computing skills. As you can see, the aggregator has the following significant advantages:

L structure is simple and easy to implement, and memory calculation can be realized quickly.

L supports multiple calling interfaces, and there are no obstacles to application integration.

L supports transparent optimization, which can significantly improve computing performance

L supports multiple data sources to facilitate the implementation of hybrid computing

The syntax is agile and exquisite, and complex business logic can be easily implemented.

With regard to memory computing, there is also a topic of multi-computer distributed computing, which will be introduced in subsequent articles.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.