The MMU slices memory in pages.
Pages sizes
32bits SPARCs support 4 page sizes: 4kB, 1MB, 256MB and 4GB, these sizes are named respectively page, segment, region and context.
All these pages are aligned.
For a 4kB page, the MMU transforms bits [31:12] of the virtual address into bits [35:12] of the physical address, bits [11:0] are unmodified.
For a 16MB page, the MMU transforms bits [31:24] of the virtual address into bits [35:24] of the physical address, bits [23:0] are unmodified.
Etc.
Large pages are used for the framebuffer, ROM, kernel… These areas are static. Dynamically allocating large pages is rarer, it can be useful for things like databases, it requires specific OS support.
Having large swaths of memory allocated together reduces MMU’s work (Fewer TLB misses, smaller pages tables).
When the CPU starts, the MMU is disabled and physical addresses are equal to virtual addresses. The 4GB page mode is a bit like that. Little use.
36 bits
The Sparc V8 reference MMU provides a 36bits physical address space, which is larger than the 32 address bits provided by the integer unit. This MMU can map each 4kB page anywhere in a 64GB address range.
You could imagine having more than 4GB of RAM, shared between several processes, each one accessing a 2GB portion. Some large many-CPUs Sun4d servers actually featured this configuration.
Practically, for our project, the 36 address bits are not used for accessing a lot of RAM. It is available here because the SparcStation-10/20 memory map spreads peripherals over the full 64GB physical address range.
A SparcStation-5 mode was added later, it only needs 31 address bits.
As the unneeded address bits are left unconnected and are discarded during FPGA compilation, the cost of these 4 additional address bits is quite limited.
Contexts
For a multitasking, multiuser OS as UNIX, it shall be possible for each process to be assigned a different memory map. SPARCs (and MIPS, ARMs) use contexts.
Contexts provide an additional indirection level, above the 4GB mapping.
Contexts are not directly related to the process IDs, as the number of hardware contexts can be lower than the number of processes. Contexts are dynamically allocated to processes. Our CPU is usually configured with 256 contexts.
The operating system kernel can access all tasks and all memory, it can also switch contexts. The kernel code must have the same mapping across all contexts, all tasks.
On SPARCs, there is no “kernel/supervisor” address space, context or memory mapping. Instead, being in supervisor mode changes the access rights and allows unrestricted access to regions out of user level software reach.
Then, it is the operating’s system role to manage virtual memory and divide the 32bits virtual address space into an application/user user and a supervisor/kernel area. RAM is usually mapped in both areas simultaneously, and when it is too large for the 32bits address space, virtual pages swapping may be necessary (see “highmem”, “PAE” issues with Linux).
(MIPS and SuperH, AVR32, … names its 8bits context register “Address Space Identifier”, which have no relation with SPARC ASIs. ARMs uses similar contexts named “ASID”. PowerPC have a “PID”: Process ID register. This is the “simple” version, MIPS32, PowerPC, ARMs MMUs are different)
TLBs
Each page have a set of characteristics: physical address, access rights…they are stored in temporary buffers, traditionally called “Translation Look-aside Buffers”. As the CPU cannot internally store all translations for all pages, the TLBs are managed as a cache of the latest accessed pages.
The replacement of TLBs contents as new pages are accessed is done either by software or through hardware mechanisms. We will see that later.
Each TLB stores the characteristics of a virtual memory area. Our MMU can currently manage up to 4 TLBs for datas and 4 TLBs for instructions (which is few).
Here is the actual content of our MMU’s TLBs (mmu_pack.vhd).
TYPE type_tlb IS RECORD v : std_logic; -- Valid va : unsigned(31 DOWNTO 12); -- Virtual Address st : unsigned(1 DOWNTO 0); -- Short Translation ctx : type_context; -- Context acc : unsigned(2 DOWNTO 0); -- Access Protection bits -------------------------------------------------------------- ppn : unsigned(35 DOWNTO 12); -- Physical Page Number c : std_logic; -- Cacheable m : std_logic; -- Modified END RECORD;
- V : Validity
- VA : Virtual Address
- ST : Short Translation
- CTX : Context
- ACC : Access Permissions
- PPN : Physical Page Number
- C : Cacheable
- M : Modified
If V=1, the TLB content is valid and contains translation information. If V=0, the TLB is empty.
The high significant bits of the virtual address are compared to this entry to determine whether the TLB matches the transfer.
For 4kB pages, addresses [31:12] are compared. For 256kB pages, addresses [31:28] are compared, etc.
Page size: 00=4kB, 01=1MB, 10=256MB, 11=4GB.
Context number. Default is 8bits. The context register is compared to this record. For supervisor pages (see ACC below), it can be ignored as the kernel resides at the same address across all contexts.
Indicates the allowed accesses for the page. See SparcV8 standard, page 248.
ACC | User | Supervisor |
---|---|---|
0 | Read Only | Read Only |
1 | Read/Write | Read/Write |
2 | Read/Execute | Read/Execute |
3 | Read/Write/Execute | Read/Write/Execute |
4 | Execute Only | Execute Only |
5 | Read Only | Read/Write |
6 | No Access | Read/Execute |
7 | No Access | Read/Write/Execute |
Types 6 and 7 indicates supervisor pages and are context-independent for the MMU (and cache).
This is the physical address associated to the page.
For selecting between cacheable and non cacheable accesses.
This bit is set if the page contents has been modified. This is unrelated to the state of cache lines (which have a Modified state in write-back configurations)
TLBs are made of three parts:
– V/VA/CTX/ST is used as content addressable memory. The MMU matches these records with the current memory access.
– ACC contents will determine if the access should be authorised or not.
– PPN/C/M generates the external access: physical address, caching.
The comparisons between the Virtual Address and each TLB can have several outcomes:
- The TLB is invalid (V=0), no information can be taken from it.
- The TLB Virtual Address (VA) mapping does not include the pending address.
- The TLB Context (CTX) does not match the current context.
- The TLB matches the address. The access control information indicates that the access is forbidden (privilege violation: no execute, no write, supervisor only…).
- The TLB matches the address. The area was never written and it is a write access (Modified Bit).
- The TLB matches the address. The access control information indicates that the access is authorised.
➜ Ignore the TLB
➜ Ignore the TLB
➜ Ignore the TLB
➜ Trigger a trap “Protection fault”. There are different traps for data and instruction faults.
➜ Update the page table in main memory before continuing. Update the TLB.
➜ Use the TLB content to generate the physical address, continue.
Obviously, for the CPU to go forward, the last case should occur with one of the TLBs for 99% of accesses.
IF no TLB match, the MMU cannot immediately decide what to do. It must find elsewhere that information. Selecting which TLB shall be updated is a problem as well. It wil be covered in following posts.
Our TLBs are either empty or contain valid address translation information. They cannot indicate that a memory area is not mapped or disabled.
All of the CPU accesses go through the MMU, even those which correspond to I/O devices. These special resources are generally only accessible to the operating system, don’t change their address often and are marked as non-cacheable.
So I guess Linux probably has suppport for PAE on sparc32? unless Linux just ignores memory above 4Gb on sun4d etc….
[Sorry for this late answer : Summertime !]
Well, I shouldn’t have mentioned PAE, which is an hack, introduced with the Pentium Pro, for extending the x86 address range beyond 32bits. SPARC’s method is a bit cleaner, the 4 extra address bits are not an extension, there were planned from the beginning in the MMU.
Linux SPARC32 does ‘HIGHMEM’ which means, AFAIK, that some pages used by the kernel to map the RAM are dynamically swapped as needed. As the 32bits virtual address space must be split and shared between the application and the kernel area (including framebuffer, io ports…), problems occurs well before RAM size reaches 4GB.
Reading Linux source code, there are a few area where the RAM size is stored in an unsigned long, so as-is it cannot support more than 4GB RAM even on Sun4d systems.
The physical RAM layout is provided in the “available” atribute of the “/memory” OpenFirmware node (arch/sparc/prom/memory.c). Each RAM module slot is given a fixed memory area. If the installed DIMM is smaller, there is a “hole” in the RAM area.
The best Sun4m computer, the SparcStation 20, only supports up to 512MB of RAM. It should be possible to access all the RAM without dealing with HIGMEM page swapping.
NetBSD also supports Sun4m and Sun4d.
The memory mapping is probed in kernel/arch/sparc/sparc/promlib.c/prom_makememarr(), total RAM is stored in a signed integer in kernel/arch/sparc/sparc/pmap.c/get_phys_mem().
An example, My SS20 (I wish I had a Cray Sun4d computer, and a lot of room !).
Memory 0..7 : 64MB + 64MB + 64MB + 32MB + 16MB + 32MB + 32MB + 16MB = 320MB RAM
Attributes :
reg
0 00000000 04000000 : 64MB
0 04000000 04000000 : 64MB
0 08000000 04000000 : 64MB
0 0C000000 01000000 : 32MB/2
0 0E000000 01000000 : 32MB/2
0 10000000 01000000 : 16MB
0 14000000 01000000 : 32MB/2
0 16000000 01000000 : 32MB/2
0 18000000 01000000 : 32MB/2
0 1A000000 01000000 : 32MB/2
0 1C000000 01000000 : 16MB
available
0 1C000000 00F51000 : A part is already used for the BIOS
0 1A000000 01000000
0 18000000 01000000
0 16000000 01000000
0 14000000 01000000
0 10000000 01000000
0 0e000000 01000000
0 00000000 0d000000
Don’t forget the Hyperstation 30 http://www.hyperstation.de/hyperSTATION_30/hyperstation_30.html
Which had 66Mhz mbus and 1Gb ram… I believe it was the true pinnacle of sun4m. Although I suppose its possible FPGAs will eventually be good enough to beat it on a budget.
The most pressing memory issue is that Linux isn’t relocatable and is rather bloated… so its not practical to build recent kernels and expect them to be small enough to boot on sun4m. Also, I received a RasterFlex-HR as well recently which should be interesting. I was wanting to get a Sun4d but they are quite hard to find… :/ in the meantime my SS20 is pretty nice especially since its easy to put a large disk it in it.
I’m considering getting a Cyclone V GX starter kit and trying to get your designs working on there… http://www.terasic.com.tw/cgi-bin/page/archive.pl?Language=English&No=830 … looks like a perfect fit even has 4Mb SRAM on top of the 512MB DDR2.
Didn’t know about the HyperStation : “Extreme Edition” Sun4m! Probably a rare item.
This Terasic board indeed is quite interesting, with a reasonable price (most FPGA eval. boards are insanely expensive !). If Altera’s Logic Elements are comparable to Xilinx’s, this FPGA seems larger than the Spartan6 LX45T (the whole SparcStation currently takes around 55% of the chip).
The only important thing that is missing is an Ethernet PHY, it should be possible to connect one through a daughterboard. Another detail is the lack of nonvolatile FLASH (for the BIOS), a small SPI flash chip could be connected to the available I/Os (RAM can also be preloaded as a temporary solution).
Of course, porting to an Altera implies some [worthwile] efforts :
– There are a few details which are somewhat optimised for Xilinx (FIFOs implementations particularly). I don’t know how it would synthesise on Altera.
– It seems to have a ready-made hardware DRAM controller (and a lot of RAM : Great !). It is very important as building one from scratch is quite complex (I once wrote a fairly simple SDRAM controller. DDR3 is dreadful.).
– There is a MicroSD slot. Writing a SD-card controller and getting rid of Xilinx’s obsolete and slow SystemACE chip for CompactFLASH would be very useful, even for the Xilinx board!
Anyway, it can be a a good board for playing with FPGAs, learning VHDL and Verilog…
I may try later this year to install the Altera dev. tools and try to compile TEMLIB.
Finally picked up that FPGA board… it got here yesterday.
So I’ll have to have a go at getting it to work with an Altera Cyclone V memory controller I guess…. hopefully sooner rather than later :). Also ordered a scsi2sd for my Sparcbook 3GX… that is going to be a lot of fun as well.
Great ! Hope you also have some spare time…
Despite the lack of recent posts, the site is not dead ! I’m working on a few tricky things, which take much more time than expected.