[Leaplist] LINUX memory shortfall (long)

[Bryan J. Smith]

[ PREFACE:

A)  I know I'm going to regret helping/publicly posting again.

B)  Please _reserve_ the common, never0ending complaints about my
"length."  This issue is a combination of no less than _at_least_ six
(6) major issues (before we even tackle the minor ones -- and I could
go on forever on this, and AMD doesn't even scratch it in its 5
volumes of manuals on the subject ;).  I'm sorry I can't "simplify
it" into 1 issue.  So, again, if you want to complain about the
length, it will just remind me why I should _not_ help people, or at
least not publicly.  Seriously, that's not my attitude, but the
attitude that ruins my desire to even help anymore.  I used to post
anyway for Google archiving and other things, let alone I got a lot
of work for them, but it's not worth it (especially as I work for
someone else now so I don't have to "find my own dinner" anymore). 
Just a consideration and reminder of why I post (it is _not_ to "look
smart" or otherwise, but to honestly explain and help).  Some things
just can't be "simplified.  ;)

C)  Of additional note, I'm on the digests, so my posts won't thread
correctly by Message-ID.

With that all said ... ]

Understand There are multiple things at work here:  

1.  32-bit is _always_ 4GiB flat

By default, 32-bit if 4GiB max, that means you can only address 4GiB
of memory contiguously, _period_.  I'll address Processor Address
Extensions (PAE) in another number, but programs _still_ have the
32-bit/4GiB "flat" limitation.

2.  You always _lose_ "physical" memory under 32-bit/4GiB

As _always_ on a 16-bit PC BIOS with 32-bit Extensions (or even
32-bit EFT firmware -- e.g., Apple Intel Macs), anywhere from
256-768MiB (0.25-0.75GiB) will be _reserved_ for memory mapped I/O
under 32-bit/4GiB.

This is why you only see 3.25-3.75GiB of RAM of "real/actual memory."

Only 64-bit EFT (Itanium, some select Intel IA-32e/EM64T servers
under test) has a firmware that does not do this, although AMD's
standard x86-64 paging will often automagically avoid this with an
x86-64 kernel, 48-bit "flat" addressed kernel aka "Long Mode" (more
follows).

3.  36-bit Processor Address Extensions (PAE)

You can use a 36-bit PAE to address more than 4GiB of memory on a
system.  Depending on the paging mechanisms of the processor, it will
"page in/out" memory from above 4GiB to below 4GiB.

This can include memory that is "lost" to memory mapped I/O -- such
as the 256-768MiB "lost" -- to above 4GiB.  This requires an PAE
kernel, like "kernel-PAE" (which includes SMP ;).

4.  Some mainboards don't let you remap memory above 4GiB

Some mainboards _refuse_ to let physical memory up to 4GiB, but that
are located in the 256-768MiB "hole" (used for memory mapped I/O
under 4GiB) to be mapped above 4GiB for usable with a PAE kernel. 
I've seen this with _most_ Intel mainboards.  Most AMD mainboards
have no such issue (although a few with poor BIOS options may).

It's really a long story, but it boils down to the fact that _all_
AMD 64-bit platforms are _physically_ capable of 40-bit addressing,
and Intel "cripples" many of its platforms or processors to a
physical 32-bit limit (virtually all Socket-478, all Celeron
processors, etc...).

5.  "Real/actual" memory and "user v. kernel space" memory

This 3.25-3.75GiB "real/actual" memory limitation is _in_addition_ to
any user v. kernel space memory models.  In other words, trying
different kernels (like one with the 4G/4G user/kernel addressing
using PAE) won't help if the mainboard doesn't allow remapping of
that "lost memory" below 4GiB above 4GiB.

If the Fedora "kernel-PAE" doesn't give you more than 3.25-3.75GiB
"real/actual" memory after fiddling with various BIOS settings,
you're SOL.  It's a mainboard limitation (or at least in how the
Linux kernel knows how to support the mainboard), not a Linux (or
Fedora) issue.

6.  Intel issues even with a "Long Mode" (52-bit PAE, IA-32e/EM64T
kernel)

Intel processors are not full AMD x86-64 (AMD64) compatible, and have
several issues when in 48-bit "flat" addressing aka "Long Mode" (PAE
emulation using 52-bit address registers for 48-bit "flat"
addressing).  When most Linux kernels detect only IA-32e (EM64T)
support, it still reserves the 256-768MiB the mainboard reserves,
leaving that "hole."  It may or may not recover the unused memory. 
This has to do with the lack of an I/O MMU in Intel processors that
safely supports (e.g., maintains processor core cache and I/O memory
coherency) above 4GiB.

AMD processors, when their I/O MMU is on (or even when portions are
disabled -- the I/O MMU is one interesting beast, especially since
Intel doesn't have one, but most things are written for Intel), does
_not_ need memory mapped I/O below 64-bit when in "flat" 48-bit
addressing aka "Long MOde" (PAE emulation using 52-bit addresses). 
So you have a _true_, _full_ 32-bit/4GiB when a 32-bit/36-bit program
runs on it.

Again, most desktop Intel mainboards, even using an IA-32e (EM64T)
kernel require this memory hole, even when running in 64-bit (48-bit
"Long Mode"), although some should be able to re-map to above 4GiB. 
It all depends.  You're far more likely to see issues with Intel
mainboards with remapping than AMD ones, because of the advantaged
paging design in AMD processors (although Core has some tricks for
remapping, even without the I/O MMU).

-- Bryan

P.S.  If you're trying to compare to Windows, do _not_ bother.  Only
NT 5.x "Server" releases (2000/2003) come with PAE kernel support. 
NT 5.x "Home/Pro" releases (2000 Pro, XP Pro, XP Home) do _not_.  So
you will see the memory hole in all cases of non-Server versions, and
even some Server versions (depending on the edition).

P.P.S.  Running a 64-bit (48-bit "Long Mode") Linux release has its
own set of issues, such as library-program incompatibility.  E.g., a
48-bit "Long Mode" (16-bit segment * 2^32 + 32-bit offset = 48-bit
"flat" address) Linux program cannot call a 32-bit library (16-bit
segment + 32-bit offset = 48-bit "segmented" address) and vice-versa.
 They are separate "memory models" for programs/libraries and are
_not_ compatible with each other, you have to have _both_ sets of
libraries to run _either_ type of program.

Most 64-bit Linux releases are all 64-bit, with a few offering select
32-bit libraries (Fedora, SuSE) -- this is called multi-lib or
multi-arch (and is a PITA sometimes).  Others just have all sorts of
issues, and hacks (like the Mozilla plug-in to run 32-bit plug-ins on
64-bit) don't work very well.  Microsoft avoids this on its 64-bit
OSes by just shipping virtually all 32-bit libraries (which makes the
OS actually slower for true 64-bit programs, or the programs
themselves must come with the 64-bit libraries -- rather stupid and
self-defeating technically, but not marketing wise).

For more on "memory models," see my old blog post here:  
http://thebs413.blogspot.com/2005/10/what-is-x86-64-long-mode-memory-model.html



-- 
Bryan J. Smith       Professional, Technical Annoyance
b.j.smith at ieee.org  http://www.linkedin.com/in/bjsmith
------------------------------------------------------
       Fission Power:  An Inconvenient Solution