[Leaplist] LINUX memory shortfall (long)

[Kevin Anderson]

This is quite interesting and eyeopening.

So I'd like to ask for some advice.

I'm looking to build a VMware server.  It'll be a whitebox, and I'm 
looking for 16G of RAM for it.  What you've written has lead me to 
believe that buying an Intel board and processors would be a mistake.  
Is that correct? 

Or am I OK with either AMD or Intel if it's a 64 bit OS?   (It'll be ESX 3).

Kev.

Bryan J. Smith wrote:
> [ PREFACE:
>
> A)  I know I'm going to regret helping/publicly posting again.
>
> B)  Please _reserve_ the common, never0ending complaints about my
> "length."  This issue is a combination of no less than _at_least_ six
> (6) major issues (before we even tackle the minor ones -- and I could
> go on forever on this, and AMD doesn't even scratch it in its 5
> volumes of manuals on the subject ;).  I'm sorry I can't "simplify
> it" into 1 issue.  So, again, if you want to complain about the
> length, it will just remind me why I should _not_ help people, or at
> least not publicly.  Seriously, that's not my attitude, but the
> attitude that ruins my desire to even help anymore.  I used to post
> anyway for Google archiving and other things, let alone I got a lot
> of work for them, but it's not worth it (especially as I work for
> someone else now so I don't have to "find my own dinner" anymore). 
> Just a consideration and reminder of why I post (it is _not_ to "look
> smart" or otherwise, but to honestly explain and help).  Some things
> just can't be "simplified.  ;)
>
> C)  Of additional note, I'm on the digests, so my posts won't thread
> correctly by Message-ID.
>
> With that all said ... ]
>
> Understand There are multiple things at work here:  
>
> 1.  32-bit is _always_ 4GiB flat
>
> By default, 32-bit if 4GiB max, that means you can only address 4GiB
> of memory contiguously, _period_.  I'll address Processor Address
> Extensions (PAE) in another number, but programs _still_ have the
> 32-bit/4GiB "flat" limitation.
>
> 2.  You always _lose_ "physical" memory under 32-bit/4GiB
>
> As _always_ on a 16-bit PC BIOS with 32-bit Extensions (or even
> 32-bit EFT firmware -- e.g., Apple Intel Macs), anywhere from
> 256-768MiB (0.25-0.75GiB) will be _reserved_ for memory mapped I/O
> under 32-bit/4GiB.
>
> This is why you only see 3.25-3.75GiB of RAM of "real/actual memory."
>
> Only 64-bit EFT (Itanium, some select Intel IA-32e/EM64T servers
> under test) has a firmware that does not do this, although AMD's
> standard x86-64 paging will often automagically avoid this with an
> x86-64 kernel, 48-bit "flat" addressed kernel aka "Long Mode" (more
> follows).
>
> 3.  36-bit Processor Address Extensions (PAE)
>
> You can use a 36-bit PAE to address more than 4GiB of memory on a
> system.  Depending on the paging mechanisms of the processor, it will
> "page in/out" memory from above 4GiB to below 4GiB.
>
> This can include memory that is "lost" to memory mapped I/O -- such
> as the 256-768MiB "lost" -- to above 4GiB.  This requires an PAE
> kernel, like "kernel-PAE" (which includes SMP ;).
>
> 4.  Some mainboards don't let you remap memory above 4GiB
>
> Some mainboards _refuse_ to let physical memory up to 4GiB, but that
> are located in the 256-768MiB "hole" (used for memory mapped I/O
> under 4GiB) to be mapped above 4GiB for usable with a PAE kernel. 
> I've seen this with _most_ Intel mainboards.  Most AMD mainboards
> have no such issue (although a few with poor BIOS options may).
>
> It's really a long story, but it boils down to the fact that _all_
> AMD 64-bit platforms are _physically_ capable of 40-bit addressing,
> and Intel "cripples" many of its platforms or processors to a
> physical 32-bit limit (virtually all Socket-478, all Celeron
> processors, etc...).
>
> 5.  "Real/actual" memory and "user v. kernel space" memory
>
> This 3.25-3.75GiB "real/actual" memory limitation is _in_addition_ to
> any user v. kernel space memory models.  In other words, trying
> different kernels (like one with the 4G/4G user/kernel addressing
> using PAE) won't help if the mainboard doesn't allow remapping of
> that "lost memory" below 4GiB above 4GiB.
>
> If the Fedora "kernel-PAE" doesn't give you more than 3.25-3.75GiB
> "real/actual" memory after fiddling with various BIOS settings,
> you're SOL.  It's a mainboard limitation (or at least in how the
> Linux kernel knows how to support the mainboard), not a Linux (or
> Fedora) issue.
>
> 6.  Intel issues even with a "Long Mode" (52-bit PAE, IA-32e/EM64T
> kernel)
>
> Intel processors are not full AMD x86-64 (AMD64) compatible, and have
> several issues when in 48-bit "flat" addressing aka "Long Mode" (PAE
> emulation using 52-bit address registers for 48-bit "flat"
> addressing).  When most Linux kernels detect only IA-32e (EM64T)
> support, it still reserves the 256-768MiB the mainboard reserves,
> leaving that "hole."  It may or may not recover the unused memory. 
> This has to do with the lack of an I/O MMU in Intel processors that
> safely supports (e.g., maintains processor core cache and I/O memory
> coherency) above 4GiB.
>
> AMD processors, when their I/O MMU is on (or even when portions are
> disabled -- the I/O MMU is one interesting beast, especially since
> Intel doesn't have one, but most things are written for Intel), does
> _not_ need memory mapped I/O below 64-bit when in "flat" 48-bit
> addressing aka "Long MOde" (PAE emulation using 52-bit addresses). 
> So you have a _true_, _full_ 32-bit/4GiB when a 32-bit/36-bit program
> runs on it.
>
> Again, most desktop Intel mainboards, even using an IA-32e (EM64T)
> kernel require this memory hole, even when running in 64-bit (48-bit
> "Long Mode"), although some should be able to re-map to above 4GiB. 
> It all depends.  You're far more likely to see issues with Intel
> mainboards with remapping than AMD ones, because of the advantaged
> paging design in AMD processors (although Core has some tricks for
> remapping, even without the I/O MMU).
>
> -- Bryan
>
> P.S.  If you're trying to compare to Windows, do _not_ bother.  Only
> NT 5.x "Server" releases (2000/2003) come with PAE kernel support. 
> NT 5.x "Home/Pro" releases (2000 Pro, XP Pro, XP Home) do _not_.  So
> you will see the memory hole in all cases of non-Server versions, and
> even some Server versions (depending on the edition).
>
> P.P.S.  Running a 64-bit (48-bit "Long Mode") Linux release has its
> own set of issues, such as library-program incompatibility.  E.g., a
> 48-bit "Long Mode" (16-bit segment * 2^32 + 32-bit offset = 48-bit
> "flat" address) Linux program cannot call a 32-bit library (16-bit
> segment + 32-bit offset = 48-bit "segmented" address) and vice-versa.
>  They are separate "memory models" for programs/libraries and are
> _not_ compatible with each other, you have to have _both_ sets of
> libraries to run _either_ type of program.
>
> Most 64-bit Linux releases are all 64-bit, with a few offering select
> 32-bit libraries (Fedora, SuSE) -- this is called multi-lib or
> multi-arch (and is a PITA sometimes).  Others just have all sorts of
> issues, and hacks (like the Mozilla plug-in to run 32-bit plug-ins on
> 64-bit) don't work very well.  Microsoft avoids this on its 64-bit
> OSes by just shipping virtually all 32-bit libraries (which makes the
> OS actually slower for true 64-bit programs, or the programs
> themselves must come with the 64-bit libraries -- rather stupid and
> self-defeating technically, but not marketing wise).
>
> For more on "memory models," see my old blog post here:  
> http://thebs413.blogspot.com/2005/10/what-is-x86-64-long-mode-memory-model.html
>
>
>
>