Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to test the minimum time interval of VxWorks response to PCIe interrupts, which has a certain reference value, and interested friends can refer to it. I hope you can learn a lot after reading this article.

Interrupts are the most common means of notifying the operating system of external events. Interrupt handling mechanism is the basis of computer multitasking environment and the guarantee of real-time performance of the system. VxWorks is an embedded real-time operating system designed and developed by Wind River Company in 1983. Kernel wind has some advantages compared with other embedded real-time operating systems in task scheduling, interrupt processing and network processing. In particular, the microsecond interrupt processing provided by it lays the foundation for VxWorks's flagship position in the field of embedded real-time operating system. In this paper, through the FPGA development board with PCIe interface, in the VxWorks6.8 version of the operating system environment, measured whether the minimum time interval of VxWorks operating system interrupt processing is the legendary microsecond level.

Preparatory work

The hardware platform environment is shown in the following figure, which is connected by two devices with Ethernet ports. One end is a FPGA development board with PCIe plugged into the PC, which runs the Windows operating system; the other end is an embedded device, which runs the VxWorks operating system.

1. Embedded devices

The motherboard is the P2020 development board, and the PCIe board is the black gold Xilinx Artix-7 PCIE AX7103 FPGA development board, which runs the VxWorks operating system.

2. PC side

A mainframe computer, take apart (the chassis is dirty, excuse me), connect to the black gold Xilinx Artix-7 PCIE AX7103 FPGA development board through the PCIe connection, run the Win7 operating system.

The two devices are connected by a double redundant network cable.

Interrupt handling flow

In the above environment, according to the requirements of Ethernet frame transmission, the interrupt processing of either end consists of three different main bodies, first, the interrupt response mechanism of the CPU kernel, then the processing of the interrupt response after adding the operating system, and then the operating system requirements, and then the PCIe hardware devices also have a set of specifications for sending interrupts to the CPU operating system. The interrupt handling mechanism of either side can be described in long words, which I will not repeat here.

The PCIe bus supports two interrupt modes, the traditional INTx interrupt and the interrupt request mechanism based on memory write request, namely message interrupt. The traditional INTx interrupt is used in the design of this paper. For the convenience of description, we describe the interrupt processing flow from the point of view of FPGA timing diagram, which is divided into host (interrupt signal sent by PCIe to host), PCIe hard core, interrupt enable signal configured by driver, and interrupt source on FPGA side. The following figure shows the interrupt interaction flow between the FPGA development board and the host when the host operating system is VxWorks.

1) there are three interrupt sources on the FPGA side that can trigger interrupts, namely, DMA write start, DMA write completion and DMA read completion interrupts. The write start interrupt source is when FPGA informs the host that there is data to be uploaded through DMA write operations; the write completion interrupt is when FPGA encapsulates all data into DMA write request packets; and the read completion interrupt means that FPGA receives all DMA read completion packets from the host. At the "1" in the picture above, the interrupt source mwr_start_interrupt has been pulled up.

2) any interrupt source is pulled up, and the PCIe IP core is configured with a "set interrupt" timing on the FPGA side. After a successful handshake between cfg_interrupt and cfg_interrupt_rdy, a high cfg_interrupt_assert is a set interrupt. (cfg_interrupt is the interrupt request sent to the host by the PCIe hard core, and cfg_interrupt_rdy is the response of the host after receiving the interrupt request. At this time, you need to look at the status of the cfg_interrupt_assert. If the cfg_interrupt_assert is high, the interrupt is set, as shown at "2" in the figure above; if the cfg_interrupt_assert is low, the interrupt request is cleared, as shown at "5" in the figure above. )

3) some time after "setting the interrupt" (about 17 clocks here), the INTA of the hard interrupt level on the host side is pulled up, and then the FPGA card really sends an interrupt to the host. Such as the "3" in the picture above.

4) after the driver detects that the interrupt level is raised, it writes 1 to the [31] bit of the control status register 04H in the BAR space of the PCIe by PIO write operation, and turns off the function of receiving the interrupt. At this time, the interrupt enable signal line int_dis_o is pulled up, such as the "4" position in the figure above. When the int_dis_o is high, the CPU no longer responds to the interrupt request of the FPGA board, which is very important here. Then CPU reads the interrupt status register of FPGA in the form of PIO read.

5) FPGA sends the value of the interrupt status register to CPU in the form of PIO read completion packet, tells CPU what kind of interrupt the interrupt is, and configures the "clear interrupt" timing. As shown at "5" in the image above.

6) after recording the interrupt source, the CPU driver resets the corresponding interrupt flag bits, as shown at "6" in the above figure. (it can also be done by FPGA itself.)

7) FPGA lowers the corresponding interrupt source signal, as shown at "7" in the figure above.

8) the CPU driver writes 0 to the [31] bit of the control status register 04H through the PIO write operation, and restarts the receiving interrupt function. As shown at "8" in the image above.

9) repeat step 1) start the next interrupt; 10) set the next interrupt sequence; 11) pull up the hard interrupt level again.

The following figure shows the interaction flow between CPU and FPGA board during a complete DMA read operation, which will finally involve the interrupt of DMA read completion, a brief description of the detailed process.

Minimum interval between VxWork responses to PCIe interrupts

In order to obtain the minimum interval of VxWorks response to PCIe interrupts, we count the clock on the FPGA side of the two "set interrupt" intervals, that is, steps 2) and 10), and reset the interval time register inter_intr_clk_cnt [31:0] count under the "set interrupt" sequence (cfg_interrupt_rdy & cfg_interrupt_assert), otherwise the count is added to one until the next "set interrupt" is reset. In this way, the time interval between two pull-up of the interrupt signal can be calculated.

In the course of testing, we used Vivado to capture two different interrupt scenarios during the actual data transmission.

1. Scenario 1: write start interrupt and read completion interrupt are processed together

With the introduction of the interrupt processing flow above, it is very convenient to analyze the waveform in the specific working state. As can be seen from the figure above, when the interrupt mrd_done_interrupt is read, the interrupt sequence is triggered, the hard interrupt level of the host is raised, the PIO operation is written to the highest bit of control and status register 04H (pictured int_dis_o signal), and the interrupt function is turned off. at this time, the interrupt timing is no longer generated on the hardware side until the driver jumps out of the interrupt reset program and writes "0" to the int_dis_o of 04H to enable interrupt. Drive PIO read interrupt status register (pictured blue line) "sample" until read completion (pictured mrd_done_interrupt signal) and write start (pictured mwr_start_interrupt signal) two interrupt flag bits are high, at this time, the driver will record and reset the two interrupt flag bits at the same time, and then the driver executes read completion interrupt and write start interrupt state machine respectively.

Scenario 2: write start interrupt and read completion interrupt are processed successively

As can be seen from the figure above, when the interrupt mrd_done_interrupt is read and the interrupt sequence is triggered, the hard interrupt level of the host is raised, and the driver writes "1" to the PIO operation of the highest bit (pictured int_dis_o signal) of the control and status register 04H, and turns off the interrupt function. At this time, the hardware side does not generate the interrupt timing until the driver jumps out of the interrupt reset program and writes "0" to the 04H int_dis_o to enable the interrupt. The driver PIO read interrupt status register (pictured blue line 1) samples to read completion (pictured mrd_done_interrupt signal). The interrupt flag bit is 1. The hardware generates a clear interrupt sequence, which lowers the hard interrupt level on the host side. Note that the write start interrupt (pictured mwr_start_interrupt signal) is just high, and the driver only records the read completion interrupt and resets the read completion interrupt flag bit (figure blue line 2). Then the driver executes the read to complete the interrupt state machine, and the driver jumps out of the interrupt state machine and then enables the interrupt again (figure blue line 3). At this time, the hardware side is allowed to generate a set interrupt sequence because the write starts to interrupt, the driver detects that the hard interrupt level signal is high again, and the driver PIO read interrupt status register (figure blue line 4) samples to the write start interrupt flag bit to be 1, and the hardware generates a clear interrupt timing. Pull down the hard interrupt level on the host side, drive the record write to start the interrupt and reset the write start interrupt flag bit (blue line 5), and then drive the execution write to start interrupting the state machine.

In the second test scenario, we can know the minimum time interval between two adjacent interrupts by counting. The hardware side generates the first interrupt segment sequence (figure blue line 1). After the first interrupt is executed, the driver side lowers the int_dis_o and enables the interrupt again. The hardware side immediately generates the interrupt sequence to perform the second interrupt operation (figure blue line 2), as shown in the following figure:

We enlarge the blue line 2 to get the following figure:

Through the adjacent interrupt clock count signal inter_intr_clk_cnt [31:0], we can know that the minimum interval between two adjacent interrupts is 365 clocks, and then it has been tested many times. The test result is 368 ~ 364. We take 365 and the clock cycle is 16ns, from which we can calculate that the minimum interrupt interval under VxWorks is 365*16=5.84us.

Conclusion: the minimum time interval of interrupt processing in VxWorks operating system is indeed the legendary microsecond level!

Interrupt response interval Test of PCIe under Windows operating system

Out of curiosity, we also tried to measure the interval between PCIe interrupt responses under the Windows operating system. The driver on Windows platform does not use the on / off interrupt enabling feature, so it only tests the interruption interval in two scenarios: VOD and copying video files.

1. Scenario 1: video on demand, with a speed of about 10Mbps

As can be seen from the above figure, after the last interrupt sequence is reset, the count of 12417491 clk (16ns) generates the interrupt sequence again, and the interrupt interval is about 198.7ms, followed by some count values: 19026416 (304.4ms), 6486433 (103.8ms), 9981793 (159.7ms). When video on demand, the bandwidth does not reach the upper limit, and the time interval between two adjacent interrupts handled by the driver is > 100ms. In order to test in the case of high bandwidth, we did the test in scenario 2.

Scenario 2: copy video at hundreds of megabits per second

As can be seen from the above figure, after the last interrupt sequence is reset, counting 4175 clk (16ns) produces the interrupt sequence again, and the interrupt interval is about 66.8us, followed by some count values: 3595 (57.5us), 7456 (119.3us), 3582 (57.3us), 4159 (66.5us). After the increase of bandwidth, the frequency of interrupts handled by the win32 driver has been significantly increased.

Problems encountered

At the beginning, the CPU operation in the interrupt processing flow did not turn on or off the operation of receiving interrupts. As a result, under the Windows platform, there was no problem. When the PCIe IP core was generated, the upper limit of the transmission bandwidth was set to 2Gbps. It was stable when testing various services through 1G Ethernet ports, and there was no operating system crash. However, in the VxWorks system testing, because the VxWorks system is very real-time and respond to interrupts in a timely manner, there will be another interrupt in the hardware when the operating system is executing an interrupt service program, which directly leads to the death of the VxWorks system, as shown in the following figure. Of course, this is also caused by the non-standard interrupt handling flow. The switch interrupt enable step will also be added to the Windows driver to test the minimum interval of PCIe interrupts corresponding to the Windows. However, from the current test data, the corresponding PCIe interrupt speed of Windows will certainly be much slower than that of VxWorks.

Thank you for reading this article carefully. I hope the article "how to test the minimum time interval between VxWorks responses to PCIe interruptions" shared by the editor will be helpful to you. At the same time, I also hope you will support us and follow the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.