[Leaplist] 32 vs 64 bit in servers

[Bryan J. Smith]

Ingo Claro <miclaro at netred.cl> wrote:
> apache with mod_php mysql tomcat

Tomcat is definitely going to be the pig here.

If you're using more than half of your concurrent userspace,
this would be 1.5GiB (3GiB userspace i686), for applications,
form/document/etc... processing, then you should be using
the x86-64 JVM instead of i686 JVM.

The i686 JVM incurs significant paging overhead as PAE is
utilized.  On the Red Hat 4G/4G kernel, it's massive (30%).

> I got lost here, sorry. I'm using centos 5.

Okay.  Memory models ...

All i486-686 processors can only address 32-bit/4GiB flat.
As such, there needs to be a "memory model" to separate
kernel (including memory mapped I/O) from userspace.

This is typically 3G user / 1G kernel.

[ SIDE NOTE:  Consumer NT i486/i686 actually uses a 2G / 2G
model, although a NTLDR switch in BOOT.INI can switch to a
3G/1G model, but there can be compatibility issues.  Only
the server versions of Windows support i686 PAE, at least
through NT 5.1 aka Server 2003. ]

Although memory above 32-bit/4GiB can be used on i686 PAE
(Processor Address Extensions), it has to be "paged" below
4GiB.  That's a good performance it.  And it gets worse.

PAE pages still need to be managed by the kernel, and that
eats kernel memory.  So well before you get to the 36-bit/
64GiB maximum of i686 PAE, you _exhuast_ all kernel memory.

[ SIDE NOTE:  Even Windows Server technical notes that only
32GiB is supported for i686. ]

To deal with this, before x86-64 became popular, Red Hat
came up with a solution.  It uses a full, flat 4GiB kernel
space called "hugemem."  This allows it to manage the full
36-bit/64GiB PAE address space (actually up to 40-bit/1TiB
on Opteron processors running i686 "hugemem" kernels).

The "catch" with Red Hat's 4G/4G model is that it pages
every time there is a system call.  For some apps, this is
killer.  At Lehman Brothers**, our core apps running on
Sun's JVM were seeing 30% performance hits with "hugemem."
As such, we were moving to x86-64 as fast as possible.

[ **NOTE:  Despite the fate of Lehman Brothers as an
investment banking firm, their datacenters live on at
Barclays.  The value of just the datacenters were almost
$2B, especially their proprietary trading system. ]

Now ...

In Red Hat Enterprise Linux (RHEL) release 5, which is what
CentOS 5 strives for bit-to-bit compatibility with, there
are only two i686 kernels -- non-PAE and PAE.  Most of the
final PAE improvements were done mid 2.6.1x (2.6.16+, RHEL
5 ships with 2.6.18), and x86-64 is now here.  I.e., RHEL
5 is based on Fedora Core 6 / Fedora 7, which was released
2006+, whereas RHEL 4 was based on Fedora Core 3, and back
in the 2004 timeframe.

So there is no i686 "hugemem" (4G/4G) kernel option in
RHEL 5.  It was removed.  I.e., Red Hat strongly encourages
anyone who needed that mode to move to x86-64.  Red Hat
also strongly encourages users of more than 4GiB (8GiB+) to
seriously consider x86-64 in general, instead of PAE.  In
fact, at this time, if you throw 64GiB (or more) at i686
PAE in RHEL 5, you're likely to kernel panic as it exhausts
memory.

[ SIDE NOTE:  There was a recent patch upstream to
artificially limit support to under 64GiB IIRC.  You should
see it in RHEL 5 shortly, although I didn't see it in even
a development 2.6.18-120.el5 kernel. ]

In other words, if you are running into issues with the
"PAE" kernel, which typically occurs at 8GiB+, you should
have already moved to x86-64.

> ok, I've got 4GiB

Then don't even bother booting i686 PAE.  Just boot the non-PAE
i686 kernel.  Yes, it will limit your memory to more like
3-3.5GiB, instead of the full 4GiB.  But PAE really does start
hammering your performance.

And if you're finding that Tomcat, MySQL, etc... are eating more
than 1.5GiB total, then you really need to be as worried about
only having 4GiB as much as going to x86-64.  i686 v. i686 PAE
v. x86-64 will be your secondary concern.

There's a lot of talk about 64-bit "performance."  From what
I've seen, GCC and even MySQL seem to give a good 20% boost
in general (before >4GiB memory considerations).  Of course,
x86-64 can eat more memory, although it really matters how your
application is written.  Most people "think" x86-64 "eats more
memory" because they are running on a desktop and loading
_both_ i486 and x86-64 libraries.  On a server, this is far
less likely.

A lot of people make a lot of common, but _incorrect_ statements
about x86-64 requiring "64-bit pointers."  In reality, _both_
x86-64 _and_ i486/i686 require 48-bit "far" pointers for heap
anyway (_below_ the "programmer level" where actual addressing
occurs).  Depending on how the application is built, pointers
are _not_ merely 64-bit or 32-bit, but many times there are
"near" and "far" and other sizes, depending on the segment
layout (even though x86-64 really doesn't have "segments," it's
still a legacy term for the sizing / "distance" of the "jump"
or to where "heap" is at).

For the most part, on 4GiB, I'd recommend i686 non-PAE.  When
you find you are using up all userspace, then consider moving
to 8GiB and x86-64.

It's up to you if you want to also test the following on 4GiB:  
- i686 PAE with the added 0.5-1.0GiB memory
- x86-64 also with the added 0.5-1.0GiB memory

I can tell you that i686 PAE will definitely be slower.  Whether
x86-64 is slower or not depends on the memory usage v. actual
improvements in execution.  Again, testing your app under _real_
_world_ load is the only way to find out.

> I've got some with core 2 duo, others with xeon.
> So, if I have 4GiB or more I should install x86_64.

In a nutshell, when you hit 8GiB, you should go x86-64.

Otherwise, at 4GiB (or less), I'd run i686 non-PAE.  Don't
bother with the i686 PAE kernel.  It's not worth it.  Even
legacy i686 apps, even enterprise apps built specifically
for PAE and require a PAE kernel, run just fine on x86-64.

After all, that's what AMD designed x86-64 to do -- 48/52-bit
flat/PAE addressing limitations for full compatibility.  It's
because even i486+ uses 48-bit virtual addressing, normalized
into a 32-bit flat address space.  Hence the name "Long Mode"
for x86-64 48-bit (52-bit with PAE enabled, which it is always
in "Long Mode").


-- 
Bryan J Smith          Professional, Technical Annoyance
b.j.smith at ieee.org    http://www.linkedin.com/in/bjsmith
--------------------------------------------------------
I don't have a "favorite Linux distro."  I use, develop
and support community efforts, often built around Linux.
Technology and solutions are my focus, not dragging in
assumptions, marketing and other concepts which dominate
non-community developed software, which I left long ago.


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.