[Leaplist] LINUX memory shortfall (long)

Thu Jan 3 00:56:02 GMT 2008

no..intel boards are fine.  If you want 16gb or ram it's really better 
to go with a 64 bit OS.

Kevin Anderson wrote:
> This is quite interesting and eyeopening.
> 
> So I'd like to ask for some advice.
> 
> I'm looking to build a VMware server.  It'll be a whitebox, and I'm 
> looking for 16G of RAM for it.  What you've written has lead me to 
> believe that buying an Intel board and processors would be a mistake.  
> Is that correct?
> Or am I OK with either AMD or Intel if it's a 64 bit OS?   (It'll be ESX 
> 3).
> 
> Kev.
> 
> Bryan J. Smith wrote:
>> [ PREFACE:
>>
>> A)  I know I'm going to regret helping/publicly posting again.
>>
>> B)  Please _reserve_ the common, never0ending complaints about my
>> "length."  This issue is a combination of no less than _at_least_ six
>> (6) major issues (before we even tackle the minor ones -- and I could
>> go on forever on this, and AMD doesn't even scratch it in its 5
>> volumes of manuals on the subject ;).  I'm sorry I can't "simplify
>> it" into 1 issue.  So, again, if you want to complain about the
>> length, it will just remind me why I should _not_ help people, or at
>> least not publicly.  Seriously, that's not my attitude, but the
>> attitude that ruins my desire to even help anymore.  I used to post
>> anyway for Google archiving and other things, let alone I got a lot
>> of work for them, but it's not worth it (especially as I work for
>> someone else now so I don't have to "find my own dinner" anymore). 
>> Just a consideration and reminder of why I post (it is _not_ to "look
>> smart" or otherwise, but to honestly explain and help).  Some things
>> just can't be "simplified.  ;)
>>
>> C)  Of additional note, I'm on the digests, so my posts won't thread
>> correctly by Message-ID.
>>
>> With that all said ... ]
>>
>> Understand There are multiple things at work here: 
>> 1.  32-bit is _always_ 4GiB flat
>>
>> By default, 32-bit if 4GiB max, that means you can only address 4GiB
>> of memory contiguously, _period_.  I'll address Processor Address
>> Extensions (PAE) in another number, but programs _still_ have the
>> 32-bit/4GiB "flat" limitation.
>>
>> 2.  You always _lose_ "physical" memory under 32-bit/4GiB
>>
>> As _always_ on a 16-bit PC BIOS with 32-bit Extensions (or even
>> 32-bit EFT firmware -- e.g., Apple Intel Macs), anywhere from
>> 256-768MiB (0.25-0.75GiB) will be _reserved_ for memory mapped I/O
>> under 32-bit/4GiB.
>>
>> This is why you only see 3.25-3.75GiB of RAM of "real/actual memory."
>>
>> Only 64-bit EFT (Itanium, some select Intel IA-32e/EM64T servers
>> under test) has a firmware that does not do this, although AMD's
>> standard x86-64 paging will often automagically avoid this with an
>> x86-64 kernel, 48-bit "flat" addressed kernel aka "Long Mode" (more
>> follows).
>>
>> 3.  36-bit Processor Address Extensions (PAE)
>>
>> You can use a 36-bit PAE to address more than 4GiB of memory on a
>> system.  Depending on the paging mechanisms of the processor, it will
>> "page in/out" memory from above 4GiB to below 4GiB.
>>
>> This can include memory that is "lost" to memory mapped I/O -- such
>> as the 256-768MiB "lost" -- to above 4GiB.  This requires an PAE
>> kernel, like "kernel-PAE" (which includes SMP ;).
>>
>> 4.  Some mainboards don't let you remap memory above 4GiB
>>
>> Some mainboards _refuse_ to let physical memory up to 4GiB, but that
>> are located in the 256-768MiB "hole" (used for memory mapped I/O
>> under 4GiB) to be mapped above 4GiB for usable with a PAE kernel. I've 
>> seen this with _most_ Intel mainboards.  Most AMD mainboards
>> have no such issue (although a few with poor BIOS options may).
>>
>> It's really a long story, but it boils down to the fact that _all_
>> AMD 64-bit platforms are _physically_ capable of 40-bit addressing,
>> and Intel "cripples" many of its platforms or processors to a
>> physical 32-bit limit (virtually all Socket-478, all Celeron
>> processors, etc...).
>>
>> 5.  "Real/actual" memory and "user v. kernel space" memory
>>
>> This 3.25-3.75GiB "real/actual" memory limitation is _in_addition_ to
>> any user v. kernel space memory models.  In other words, trying
>> different kernels (like one with the 4G/4G user/kernel addressing
>> using PAE) won't help if the mainboard doesn't allow remapping of
>> that "lost memory" below 4GiB above 4GiB.
>>
>> If the Fedora "kernel-PAE" doesn't give you more than 3.25-3.75GiB
>> "real/actual" memory after fiddling with various BIOS settings,
>> you're SOL.  It's a mainboard limitation (or at least in how the
>> Linux kernel knows how to support the mainboard), not a Linux (or
>> Fedora) issue.
>>
>> 6.  Intel issues even with a "Long Mode" (52-bit PAE, IA-32e/EM64T
>> kernel)
>>
>> Intel processors are not full AMD x86-64 (AMD64) compatible, and have
>> several issues when in 48-bit "flat" addressing aka "Long Mode" (PAE
>> emulation using 52-bit address registers for 48-bit "flat"
>> addressing).  When most Linux kernels detect only IA-32e (EM64T)
>> support, it still reserves the 256-768MiB the mainboard reserves,
>> leaving that "hole."  It may or may not recover the unused memory. 
>> This has to do with the lack of an I/O MMU in Intel processors that
>> safely supports (e.g., maintains processor core cache and I/O memory
>> coherency) above 4GiB.
>>
>> AMD processors, when their I/O MMU is on (or even when portions are
>> disabled -- the I/O MMU is one interesting beast, especially since
>> Intel doesn't have one, but most things are written for Intel), does
>> _not_ need memory mapped I/O below 64-bit when in "flat" 48-bit
>> addressing aka "Long MOde" (PAE emulation using 52-bit addresses). So 
>> you have a _true_, _full_ 32-bit/4GiB when a 32-bit/36-bit program
>> runs on it.
>>
>> Again, most desktop Intel mainboards, even using an IA-32e (EM64T)
>> kernel require this memory hole, even when running in 64-bit (48-bit
>> "Long Mode"), although some should be able to re-map to above 4GiB. It 
>> all depends.  You're far more likely to see issues with Intel
>> mainboards with remapping than AMD ones, because of the advantaged
>> paging design in AMD processors (although Core has some tricks for
>> remapping, even without the I/O MMU).
>>
>> -- Bryan
>>
>> P.S.  If you're trying to compare to Windows, do _not_ bother.  Only
>> NT 5.x "Server" releases (2000/2003) come with PAE kernel support. NT 
>> 5.x "Home/Pro" releases (2000 Pro, XP Pro, XP Home) do _not_.  So
>> you will see the memory hole in all cases of non-Server versions, and
>> even some Server versions (depending on the edition).
>>
>> P.P.S.  Running a 64-bit (48-bit "Long Mode") Linux release has its
>> own set of issues, such as library-program incompatibility.  E.g., a
>> 48-bit "Long Mode" (16-bit segment * 2^32 + 32-bit offset = 48-bit
>> "flat" address) Linux program cannot call a 32-bit library (16-bit
>> segment + 32-bit offset = 48-bit "segmented" address) and vice-versa.
>>  They are separate "memory models" for programs/libraries and are
>> _not_ compatible with each other, you have to have _both_ sets of
>> libraries to run _either_ type of program.
>>
>> Most 64-bit Linux releases are all 64-bit, with a few offering select
>> 32-bit libraries (Fedora, SuSE) -- this is called multi-lib or
>> multi-arch (and is a PITA sometimes).  Others just have all sorts of
>> issues, and hacks (like the Mozilla plug-in to run 32-bit plug-ins on
>> 64-bit) don't work very well.  Microsoft avoids this on its 64-bit
>> OSes by just shipping virtually all 32-bit libraries (which makes the
>> OS actually slower for true 64-bit programs, or the programs
>> themselves must come with the 64-bit libraries -- rather stupid and
>> self-defeating technically, but not marketing wise).
>>
>> For more on "memory models," see my old blog post here:  
>> http://thebs413.blogspot.com/2005/10/what-is-x86-64-long-mode-memory-model.html 
>>
>>
>>
>>
>>   
> _______________________________________________
> Leaplist mailing list
> Leaplist at leap-cf.org
> http://lists.leap-cf.org/mailman/listinfo/leaplist
> 

-- 
My "Foundation" verse:
Isa 54:17  No weapon that is formed against thee shall prosper; and 
every tongue that shall rise against thee in judgment thou shalt 
condemn. This is the heritage of the servants of the LORD, and their 
righteousness is of me, saith the LORD.

-- carpe ductum -- "Grab the tape"
CDTT (Certified Duct Tape Technician)

Linux user #322099
Machines:
206822
256638
276825
http://counter.li.org/