View source for Memory System

This page documents Kermit memory system internals.

The [https://www.ti.com/lit/pdf/spruf98 OMAP35x Technical Reference Manual] provides information about the SonicsMX® interconnect used in Kermit ES1. Datasheets for some Microchip PIC32 microcontrollers (e.g. [https://www.microchip.com/en-us/solutions/displays/embedded-graphics-solutions/mcu-guided-selection-tool-for-graphics/pic32mz-ef-series PIC32MZ] family) provide information about the SonicsSX® interconnect used in Kermit ES2+.

=== Glossary ===
* SMX
Abbreviation of SonicsMX®.
* SSX
Abbreviation of SonicsSX®.
* Module
Any device connected to the memory system, such as ARM cores, DMA controllers, LPDDR2...
Modules can be initiators, targets or both.
* Initiator
A module that can initiate read and write requests to the interconnect (e.g. ARM cores, DMA, ...).
* Target
A module that can only respond to requests from the interconnect. Targets may also be able to generate out-of-band signals such as interrupts.
* Agent
The connection between a module's port and the interconnect. If a module has multiple ports (e.g. an Initiator and Target module), each port is connected to the interconnect by a different agent (e.g. one IA and one TA).
* TA
Short for Target Agent. The agent connecting an Initiator module to the interconnect.
* IA
Short for Initiator Agent. The agent connecting a Target module to the interconnect.
* OCP
[https://www.ocpip.org/ Open-core Protocol], a standard point-to-point protocol allowing communication between a master and a slave port.
* OCP Master Port
A port that can generate OCP commands. Initiators always include at least one master port.
* OCP Slave Port
A port that can respond to OCP commands. Targets always include one slave port.
* Interconnect
Logic device enabling connection between initiator and target modules connected to it.
* Out-of-band error
OCP signal associated to a device error-reporting scheme.
This is opposed to an in-band error, which is associated to the protocol's error-reporting scheme.
Out-of-band errors are also known as ''sideband'' errors.

== Generalities ==

The bus fabric of Kermit is made of several SonicsSX® interconnects on which peripherals are connected.

All communication is done using the OCP protocol; however, adapters (e.g., AXI2OCP for Cortex-A9) may be present at Agent level.

There are three revisions of the memory system layout:
* Original layout for Kermit1.0 ES1 - uses the SMX interconnect (as found in BeagleBoard SoC) instead of SSX
* Second revision for Kermit1.0 ES2 - very close to the final revision, uses the SSX interconnect
* Third and final revision for Kermit1.0 ES3 and later

In the following sections, you may see mentions of register names such as <code>BEBT</code> and <code>SBEBT</code>. Unless otherwise specified, a register prefixed with <code>S</code> is merely the ARM Secure variant of the register e.g. <code>SBEBT</code> is <code>BEBT</code> for ARM Secure. Everything that applies to a register should apply to its Secure variant, except that the register is updated by Secure bus transactions, instead of Non-Secure bus transactions.

=== Signals ===

==== MReqInfo ====

<code>MReqInfo</code> is 16 bits wide in Kermit.

Bits <6:0> contain the <code>Bus Master ID</code> of initiator.
This is sometimes used for access checks, or to identify transaction initiator when a bus error occurs.

== Interconnect registers ==

There are register blocks associated to each interconnect in the system. These blocks are always 1KiB (0x400 bytes), and the registers inside them are always 64-bit.

=== Location ===

{| class="wikitable" style="text-align: center"
|-
! Address !! Name !! UIID
|-
| 0xEC000000 || Main Xbar || 0
|-
| 0xEC100000 || Center Xbar || 1
|-
| 0xEC200000 || Video Xbar || 2
|-
| 0xEC300000 || IFTU Xbar || 3
|-
| 0xEC400000 || VIP Xbar || 4
|-
| 0xEC500000 || Debug Bus || 5
|-
| 0xEC600000 || DMAC Xbar || 6
|-
| 0xED000000 || IO Slave Bus || 8
|-
| 0xED100000 || IO Master Bus || 9
|-
| 0xED200000 || Pervasive Bus || 10
|-
| 0xED300000 || Misc IO Bus || 11
|-
| 0xED400000 || North IO Master Bus || 12
|-
| 0xED500000 || Audio Bus || 13
|-
| 0xED600000 || North IO Slave Bus || 14
|-
| 0xED700000 || SRC Bus || 15
|-
| 0xED800000 || GPU Register Bus || 16
|-
| 0xED900000 || IFTU Register Bus || 17
|-
| 0xEE000000 || Camera Slave Bus || 24
|-
| 0xEE100000 || Camera Master Bus || 25
|-
| 0xEE200000 || USB Slave Bus || 26
|-
| 0xEE300000 || USB Master Bus || 27
|-
| 0xEE400000 || SubLCD Slave Bus || 28
|-
| 0xEE500000 || SubLCD Master Bus || 29
|-
| 0xEE600000 || Camera Register Bus || 30
|-
| 0xE00C0000 || Cry/SBL Bus || 48
|-
|}

The <code>UIID</code> column indicates the '''U'''nique '''I'''nterconnect '''ID'''.
Bit <code>N</code> of BEBT register indicates an error is pending on interconnect with UUID <code>N</code>.

Each interconnect register group is divided as follows:
{| class="wikitable" style="text-align: center"
|-
! Address mask !! Sub-block !! Description
|-
| <code>0xEoo00000</code> || ? || ?Sideband Interconnect?
|-
| <code>0xEoo1xx00</code> || ? || IA-related (Only present if IA exists at <code>0xEoo4xx00</code>)
|-
| <code>0xEoo30000</code> || ? || ?
|-
| <code>0xEoo30400</code> || ? || ?
|-
| <code>0xEoo3C000</code> || ? || ?
|-
| <code>0xEoo4xx00</code> || IA || Initiator Agent - one block for each initiator on the interconnect. Cross-interconnect IAs usually start at offset 0x8000.
|-
| <code>0xEoo5xx00</code> || TA || Target Agent - one block for each target on the interconnect. Cross-interconnect TAs usually start at offset 0x8000.
|-
| <code>0xEoo6xx00</code> || PM || Protection Mechanism - used to filter access to specific targets. If present, register block at <code>0xEoo6xx00</code> protects TA <code>0xEoo5xx00</code>.
|-
| <code>0xEoo6C000</code> || Interconnect PM || Protection Mechanism ?for the interconnect's RT?
|-
| <code>0xEoo70000</code> || RT || Register Target - used to interact with the interconnect itself.
|}

Note: <code>Cry/SBL Bus</code> does not follow this scheme.

=== Agent registers ===

IA and TA register blocks share a similar layout.

(List of physical address for all these interfaces can be found on [[Physical Memory]] page - names start with <code>SSX_</code> - will be migrated here at some point)

{| class="wikitable"
|+ Agent registers description
|-
! Name !! Offset !! Usage/Notes
|-
| <code>COMPONENT</code> || 0x0
| Bits [31:16] contain component code (<code>CODE</code>)

Bits [15:0] contain component revision (<code>REV</code>)

?what is ''component'' referring to? SSX?
|-
| <code>CORE</code> || 0x18
| Bits [47:32] contain vendor code (<code>VENDOR_CODE</code>)

Bits [31:16] contain core code (<code>CORE_CODE</code>)

Bits [15:0] contain revision code (<code>REV_CODE</code>)
|-
| <code>AGENT_CONTROL</code> || 0x20 || ?
|-
| <code>AGENT_STATUS</code> || 0x28 || Bit 0x0100_0000 indicates an error
|-
| <code>AGENT_ERROR_LOG</code> || 0x58 ||
|-
| <code>AGENT_ERROR_LOG_ADDR</code> || 0x60 ||
|-
| ? || 0x100 || Seen values: 1
|}

=== Register Target ===

Hosts configuration and status registers of the interconnect itself.

{| class="wikitable"
|+ RT registers description
|-
! Name !! Offset !! Usage/Notes
|-
| <code>COMPONENT</code> || 0x0 || See Agent registers
|-
| <code>NETWORK</code> || 0x10 || Bits 63:48 (63:32?) hold the Unique Interconnect ID
|-
| <code>INITID_READBACK</code> || 0x70 || Bits <7:0> contain the Initiator ID of the reader
|-
| <code>NETWORK_CONTROL</code> || 0x78 || Bits <10:8> contain TIMEOUT_BASE
|}

Note: the initiator ID is an interconnect-level concept, and unrelated to the Bus Master ID. ?Every IA has its own IID?

=== Sideband Interconnect ===

Each interconnect hosts a register block called '''S'''ideband '''I'''nterconnect, used for error reporting.

{| class="wikitable"
|+ SI registers description
|-
! Name !! Offset !! Usage/Notes
|-
| <code>SI_FLAG_STATUS_0</code> || 0x110 || Sideband Interconnect error status 0
|-
| <code>SI_FLAG_STATUS_1</code> || 0x130 || Sideband Interconnect error status 1
|-
| <code>SI_FLAG_STATUS_2</code> || 0x150 || Sideband Interconnect error status 2
|-
| <code>SI_FLAG_STATUS_3</code> || 0x170 || Sideband Interconnect error status 3
|-
| <code>SI_FLAG_STATUS_4</code> || 0x190 || Sideband Interconnect error status 4
|}

+ many registers not documented in OMAP35x TRM

=== Protection Mechanism ===

Each Target Agent may be equipped with a Firewall (also called '''P'''rotection '''M'''echanism), which can be used to restrict or filter access to it based on various criteria.

{| class="wikitable"
|+ PM registers location
|-
! Base address !! Protected TA
|-
| 0xEC060000? || <code>Spad_32KiB</code>?
|-
| 0xEC060400 || <code>LPDDR0</code> (Main LPDDR2 + Controller I/F)
|-
| 0xEC160000? || <code>Compati_2MiB</code>?
|-
| 0xEC160400? || <code>DevNull</code>?
|-
| 0xEC260000? || <code>Spad_128KiB</code>?
|-
| 0xEC560000? || <code>SDIO0</code>?
|-
| 0xEC560400? || <code>SDIO1</code>?
|-
| 0xEC560800? || <code>LPDDR1</code>? (TOOL LPDDR2 + Controller I/F)
|}

== Bus Errors ==

When a bus error occurs, an interrupt is delivered to ARM Secure (Interrupt ID 0x21=33) or Non-Secure (Interrupt ID 0x20=32).
The OS considers all bus errors to be fatal. The interrupt handlers installed for these IDs perform a register dump then stop the system.

It appears that device-initiated bus errors (e.g. DMAC) are always routed to Non-Secure interrupt. There may be other rules to consider (e.g. some register to set whether a device is Secure/Non-Secure).

It is unknown whether or not CMeP can receive bus error interrupts or how CMeP fits in the memory system.

There are two kind of bus errors: ''Internal Bus'' error and ''Target Device'' error.
An Internal Bus error occurs when the memory system fails to deliver a request to a target device - for example, trying to access non-existent memory.
A Target Device error occurs when a device successfully receives a request but is unable to handle it - for example, accessing the non-existent part of a device's memory.

=== Bus Error Attribute ===

The bus error attribute is a 32-bit value that can be recovered in a per-device MMIO register along with the bus error address.

The attribute holds multiple informations: which agent was the bus master when the error occurred, what bus command was ongoing, and, when available, the reason of the bus error.

{| class="wikitable"
|+ Bus error attribute meaning
|-
! Bitmask !! Name !! Notes
|-
| <code>0xFF80_0000</code> || <code>MReqInfo<15:7></code> || Bits <15:7> of MReqInfo signal
|-
| <code>0x003F_0000</code> || master || Bus master ID (bits <6:0> of MReqInfo signal)
|-
| <code>0x0000_0700</code> || cmd || MCmd signal
|-
| <code>0x0000_0008</code> || Burst Access Error || 
|-
| <code>0x0000_0004</code> || Register Permission Error || 
|-
| <code>0x0000_0001</code> || Address Hole Error || If set, the error occurred because the address was invalid.
|}

Sometimes, this register is not present but separated in multiple (e.g., in REGBUS).

Note that for some devices (Spad32K, Spad128K, Compati SRAM), the only valid attribute is <code>0x1</code>, indicating an invalid address.

To decode the meaning of <code>master</code> and <code>cmd</code>, shift them by 16 and 8 respectively to obtain a value between 0-63/0-7 and use the following tables:
{| class="wikitable mw-collapsible mw-collapsed"
|+ <code>MCmd</code> table
|-
! Value !! Command
|-
| 0 || Idle
|-
| 1 || Write
|-
| 2 || Read
|-
| 3 || ReadEx
|-
| 4 || ReadLinked
|-
| 5 || WriteNonPost
|-
| 6 || WriteConditional
|-
| 7 || Broadcast
|}

{| class="wikitable mw-collapsible mw-collapsed"
|+ Bus Master IDs (<code>master</code> decoding table)
|-
! Value !! Command
|-
| 0 || Reserved
|-
| 1 || ARM Core0
|-
| 2 || ARM Core1
|-
| 3 || ARM Core2
|-
| 4 || ARM Core3
|-
| 5 || ARM L2
|-
| 6 || Reserved
|-
| 7 || Reserved
|-
| 8 || Reserved
|-
| 9 || Reserved
|-
| 10 || Reserved
|-
| 11 || Reserved
|-
| 12 || Reserved
|-
| 13 || Reserved
|-
| 14 || Reserved
|-
| 15 || Reserved
|-
| 16 || DMAC0
|-
| 17 || DMAC1
|-
| 18 || DMAC2
|-
| 19 || DMAC3
|-
| 20 || DMAC4
|-
| 21 || DMAC5
|-
| 22 || Reserved (Bigmac?)
|-
| 23 || GPU
|-
| 24 || Venezia
|-
| 25 || VIP vdpd
|-
| 26 || VIP vdpm
|-
| 27 || VIP bap
|-
| 28 || Reserved
|-
| 29 || IFTU0a
|-
| 30 || IFTU0b
|-
| 31 || IFTU1a
|-
| 32 || IFTU1b
|-
| 33 || IFTU2
|-
| 34 || Reserved (<code>Compati port0</code> in ES1 [[SceKernelBusError]])
|-
| 35 || Reserved (<code>Compati port0</code> in ES1 [[SceKernelBusError]])
|-
| 36 || USB1 Host EHCI
|-
| 37 || USB1 Host OCHI
|-
| 38 || Reserved (CMeP?)
|-
| 39 || Sensor in 0
|-
| 40 || Sensor in 1
|-
| 41 || Reserved (<code>Sensor in 2</code> in ES1 [[SceKernelBusError]])
|-
| 42 || LCD DMAC
|-
| 43 || Performance Monitor
|-
| 44 || USB2 Device DMAC
|-
| 45 || Sub LCD
|-
| 46 || DMAC6
|-
| 47 || USB0 Host EHCI
|-
| 48 || USB0 Host OHCI
|-
| 49 || USB2 Host EHCI
|-
| 50 || USB2 Host OHCI
|-
| 51 || USB0 Device DMAC
|-
| 52 || Reserved (<code>USB1 Device DMAC1</code> in ES1/ES2)
|-
| 53 || Reserved (<code>USB1 Device DMAC2</code> in ES1/ES2)
|-
| 54 || Reserved (<code>USB1 Device DMAC3</code> in ES1/ES2)
|-
| 55 || USB1 Device DMAC
|-
| 56 || Reserved (<code>USB2 Device DMAC1</code> in ES1/ES2)
|-
| 57 || Reserved (<code>USB2 Device DMAC2</code> in ES1/ES2)
|-
| 58 || Reserved (<code>USB2 Device DMAC3</code> in ES1/ES2)
|-
| 59 || SD/HSMMC0
|-
| 60 || SD/HSMMC1
|-
| 61 || SD/HSMMC2
|-
| 62 || SD/HSMMC3
|-
| 63 || Memory Stick
|}

=== Clear TA error ===

In old System Software versions (e.g. 0.920 - pre-ES3, so this may no longer be valid), there exists inside the bus error module a function named <code>BusErrorClearTA</code> which works the following way.

First, choose a bus/XBar that will serve as start point (MainXBar for ES2).
Second, recursively build a list of all TAs connected to this start point. Repeat the process for all TAs that are busses or XBars.
Third, walk the obtained tree. Check the <code>STATUS</code> of all TAs. If an error is present, repeat the procedure recursively (if bus/XBar) then clear the error.

To detect an error, check if bit <code>0x01000_0000ull</code> is present in <code>STATUS</code>
To clear the error, simply write <code>0x01000_0000ull</code> to <code>STATUS</code>.

NOTE: old System Software versions do not have informations about IAs, but only TAs. This could explain the routine's name, but also means this procedure may also work/be needed on IA side.

== Miscellaneous ==

The memory system is able to distinguish if a transaction is originating from ARM Secure state or Non-Secure state because ARM Cortex-A9 processors with Security Extensions have [https://developer.arm.com/documentation/den0013/d/Security/TrustZone-hardware-architecture a bit indicating whether the access is Secure or Non-Secure added to all memory system transactions].

Attempting to perform a DMAC memcpy() from Secure to Non-Secure LPDDR0 region results in a NS bus error. The current hypothesis is that all devices on the Kermit bus diagram that have an IA are treated as ARM NS.