Performance Summary:

Performance Details:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
•   HP Power Regulator: HP Static High Performance Mode
•   HP Power Profile: Maximum performance
•   Thermal configuration: Increased Cooling
•   CPU C3-state enabled
•   Processor Power and Utilization Montoring disabled

Storage Notes

• All storages connected via 2 x 8Gb Fibre Channel links through 2 x StorageWorks 8/24 SAN switches
• to 15 x StorageWorks P2000 storage enclosures.
• Each P2000 contained 24 x 146 GB SFF 15K SAS drives.
• The enclosures were partitioned and used as follows:
•   12 x P2000 - vdisk1: RAID 10 (12 drives), 3 dbservers and 3 appservers; vdisk2: RAID 10 (12 drives), 3 infraservers and 3 webservers; vdisk3: RAID 10 (12 drives), 3 mailservers and 3 idleservers
•   3 x P2000 - vdisk1: RAID 10 (12 drives), 4 dbservers and 4 appservers; vdisk2: RAID 10 (12 drives), 4 infraservers and 4 webservers; vdisk3: RAID 10 (12 drives), 4 mailservers and 4 idleservers
• 2 x 146 GB SFF 15K SAS internal drives and 2 x 73 GB SFF 15K SAS internal drives were used for OS

Virtualization Software Notes

• Boot parameters set: console=tty0 console=ttyS1,115200 selinux=0 audit=0 elevator=deadline clocksource=tsc idle=halt intel_iommu=on
• The following services were disabled:
•   abrtd, atd, auditd, autofs, avahi-daemon, cgconfig, httpd, iptables, ip6tables, irqbalance, iscsi, iscsid, ksm, ksmtuned, libvirt-guests, libvirtd, messagebus, NetworkManager, nfslock, postfix, rhnsd, rpcbind, rpcgssd, rpcidmapd, cpuspeed
• /etc/sysctl.conf was updated with the following lines:
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.core.netdev_max_backlog = 500000
•   kernel.sched_min_granularity_ns = 10000000
•   kernel.sched_wakeup_granularity_ns = 15000000
•   kernel.sched_shares_ratelimit = 10000000
•   fs.aio-max-nr = 70656
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
• Network configuration is as follows:
•   The first integrated NC375i NIC port(eth0) is for host access
•   The 4 Solarflare NIC ports were assigned to bridges (eth10 to br0, eth11 to br1, eth12 to br2, eth13 to br3)
•   All the public VNICs of the tiles use the 4 bridges.
•   The Solarflare NICs were configured with
•   sfboot sriov=enabled vf-count=60 vf-msix-limit=1
•   All the webservers, infraservers, mailservers and appservers were assigned with a VF
• All irqs of the VFs were bound to the CPU 31-38,71-78,21-28,61-68,11-18,51-58,1-8,41-48 by round-robin
• All irqs of the HBAs were bound to CPU 71 72 73 74 by round-robin
• All irqs of the eth10 were bound to CPU 1-8,41-48
• All irqs of the eth11 were bound to CPU 11-18,51-58
• All irqs of the eth12 were bound to CPU 21-28,61-68
• All irqs of the eth13 were bound to CPU 31-38,71-78
• All VMs except for the idleservers were created with the following command line format:
•   /usr/libexec/qemu-kvm -name $name -M rhel6.1.0 -enable-kvm
•   -m $mem -mem-prealloc
•   -cpu qemu64,+x2apic -no-kvm-pit-reinjection
•   -smp $cpu,sockets=1,cores=$cpu,threads=1
•   -drive file=$diskfile,if=virtio,boot=on,cache=none,aio=native
•   -netdev tap,id=$vnic1,ifname=$vnic1,script=/etc/libvirt/qemu-ifup-spec-br${no},vhost=on
•   -device virtio-net-pci,netdev=$vnic1,mac=$mac1
•   -vnc 0.0.0.0:$vncport
•   -pcidevice host=$vfid
•   -daemonize
• idleservers and dbservers were started without the parameter: -pcidevice host=$vfid
• Web App Db and Idle VMs were launched with nice utility (-n -1 -7 -6 19) to adjust their scheduling priority
• All VMs of tiles 1,2,3,4,5,6,7,8,9,10,11,12 were bound to NUMA node 0 and prefer to allocate memory from node 0
• All VMs of tiles 13,14,15,16,17,18,19,20,21,22,23,24 were bound to NUMA node 1 and prefer to allocate memory from node 1
• All VMs of tiles 25,26,27,28,29,30,31,32,33,34,35,36 were bound to NUMA node 2 and prefer to allocate memory from node 2
• All VMs of tiles 37,38,39,40,41,42,43,44,45,46,47,48 were bound to NUMA node 3 and prefer to allocate memory from node 3

Web Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
•   abrtd, atd, auditd, autofs, cpuspeed, cups, dovecot, ip6tables, iptables, irqbalance, kdump, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcbind, rpcgssd, rpcidmapd
• Following items were added in /etc/sysctl.conf
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.neigh.default.gc_thresh1 = 4096
•   net.ipv4.neigh.default.gc_thresh2 = 8192
•   net.ipv4.neigh.default.gc_thresh3 = 8192
•   net.ipv4.neigh.default.base_reachable_time = 86400
•   net.ipv4.neigh.default.gc_stale_time = 86400
•   net.ipv6.neigh.default.gc_thresh1 = 4096
•   net.ipv6.neigh.default.gc_thresh2 = 8192
•   net.ipv6.neigh.default.gc_thresh3 = 8192
•   net.ipv6.neigh.default.base_reachable_time_ms = 86400
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
• root file-system options: barrier=0,noatime
• The following items were set in /opt/common_scripts/ostune_misc.sh which was executed after boot up
•   net.ipv4.tcp_syn_retries=4
•   net.ipv4.tcp_retries2=7
•   net.ipv4.tcp_keepalive_time=120
•   vm.swappiness=0
•   vm.overcommit_memory=1
•   net.ipv4.tcp_tw_reuse=1
• httpd.conf changes
•   Timeout 20
•   KeepAlive on
•   MaxKeepAliveRequest 0
•   KeepAliveTimeout 60
•   StartServers 1024
•   MinSpareServers 0
•   MaxSpareServers 1024
•   ServerLimit 1024
•   MaxClients 1024
•   MaxRequestsPerChild 0
•   CustomLog logs/access_log common
• NFS mount options:
•   noatime,actimeo=14400,nocto
• xnap module option used: rx_irq_mod_usec=2000

Application Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
•   abrtd, atd, auditd, autofs, cpuspeed, cups, dovecot, httpd, ip6tables, iptables, irqbalance, kdump, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcbind, rpcgssd, rpcidmapd
• Following items were added in /etc/sysctl.conf
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.neigh.default.gc_thresh1 = 4096
•   net.ipv4.neigh.default.gc_thresh2 = 8192
•   net.ipv4.neigh.default.gc_thresh3 = 8192
•   net.ipv4.neigh.default.base_reachable_time = 86400
•   net.ipv4.neigh.default.gc_stale_time = 86400
•   net.ipv6.neigh.default.gc_thresh1 = 4096
•   net.ipv6.neigh.default.gc_thresh2 = 8192
•   net.ipv6.neigh.default.gc_thresh3 = 8192
•   net.ipv6.neigh.default.base_reachable_time_ms = 86400
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
• root file-system options: barrier=0
• The following items were set in /opt/common_scripts/ostune_misc.sh which was executed after boot up
•   sysctl -w vm.swappiness=0
•   sysctl -w vm.overcommit_memory=1
•   echo 658 > /proc/sys/vm/nr_hugepages
•   id -g spec > /proc/sys/vm/hugetlb_shm_group
•   mount -t hugetlbfs hugetlbfs /dev/hugepages
• The following lines were added to glassfish domain.xml:
•   To jdbc-connection-pool line parameter 'max-pool-size=64' was set
•   To jdbc-connection-pool line parameter 'steady-pool-size=64' was set
•   To request-processing line parameter 'initial-thread-count=90' was set
•   To request-processing line, parameter 'thread-count=90' was set
•   To ejb-container line, parameter 'max-pool-size=64' was set
•   To mdb-container line, parameter 'max-pool-size=64' was set
•   In keep-alive section: parameter 'timeout-in-seconds=300' was set
•   In http-thread-pool section: parameter 'max-thread-pool-size=32' was set
•   In jvm-options section: set -XX:+UseLargePages, set -Xmn600m, set -Xms1124m, set -Xmx1124m
•   set -XX:-UseBiasedLocking, -XX:CompileThreshold=1000, -XX:+AggressiveOpts, -XX:+UseCompressedOops
•   set -XX:+UseFastAccessorMethods, -XX:+UseStringCache, -XX:+OptimizeStringConcat
•   set -Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.EPollSelectorProvider
• xnap module option used: rx_irq_mod_usec=4000

Mail Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
•   abrtd, atd, auditd, autofs, cpuspeed, cups, httpd, ip6tables, iptables, irqbalance, kdump, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcbind, rpcgssd, rpcidmapd
• Following items were added in /etc/sysctl.conf
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.neigh.default.gc_thresh1 = 4096
•   net.ipv4.neigh.default.gc_thresh2 = 8192
•   net.ipv4.neigh.default.gc_thresh3 = 8192
•   net.ipv4.neigh.default.base_reachable_time = 86400
•   net.ipv4.neigh.default.gc_stale_time = 86400
•   net.ipv6.neigh.default.gc_thresh1 = 4096
•   net.ipv6.neigh.default.gc_thresh2 = 8192
•   net.ipv6.neigh.default.gc_thresh3 = 8192
•   net.ipv6.neigh.default.base_reachable_time_ms = 86400
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
•   root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• Contents of dovecot configuration file /etc/dovecot/dovecot.conf:
• dict {
• }
• !include conf.d/*.conf
• base_dir = /var/run/dovecot/
• disable_plaintext_auth = no
• log_path = /var/log/dovecot
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mbox_write_locks = fcntl
• passdb {
• args = scheme=plain /etc/dovecot/passwd.dovecot
• driver = passwd-file
• }
• postmaster_address = [email protected]
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_limit = 1024
• }
• ssl = no
• ssl_cert =
• ssl_key =
• userdb {
• driver = passwd
• }
• xnap module option used: rx_irq_mod_usec=4000

Database Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
•   abrtd, atd, auditd, autofs, cpuspeed, cups, dovecot, httpd, ip6tables, iptables, irqbalance, kdump, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcbind, rpcgssd, rpcidmapd
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.neigh.default.gc_thresh1 = 4096
•   net.ipv4.neigh.default.gc_thresh2 = 8192
•   net.ipv4.neigh.default.gc_thresh3 = 8192
•   net.ipv4.neigh.default.base_reachable_time = 86400
•   net.ipv4.neigh.default.gc_stale_time = 86400
•   net.ipv6.neigh.default.gc_thresh1 = 4096
•   net.ipv6.neigh.default.gc_thresh2 = 8192
•   net.ipv6.neigh.default.gc_thresh3 = 8192
•   net.ipv6.neigh.default.base_reachable_time_ms = 86400
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
•   root file-system options: barrier=0
•   datastore file-system options: noatime,barrier=0,data=writeback
• Database tuning options in /var/lib/pgsql/9.0/data/postgresql.conf
•   wal_buffers = 20MB
•   checkpoint_completion_target = 0.9
•   fsync = off
•   synchronous_commit = off
•   wal_sync_method = fdatasync
•   effective_cache_size = 1200MB
•   shared_buffers = 1000MB
•   work_mem = 64MB
•   maintenance_work_mem = 1MB
•   max_connections = 250
•   ssl = off
•   full_page_writes = off
•   debug_assertions = false
•   log_min_duration_statement = 1000
•   default_statistics_target = 100
•   log_rotation_size = 16MB
•   log_destination = syslog

Infraserver VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
•   abrtd, atd, auditd, autofs, cpuspeed, cups, dovecot, httpd, ip6tables, iptables, irqbalance, kdump, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd
• Following items were added in /etc/sysctl.conf
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.neigh.default.gc_thresh1 = 4096
•   net.ipv4.neigh.default.gc_thresh2 = 8192
•   net.ipv4.neigh.default.gc_thresh3 = 8192
•   net.ipv4.neigh.default.base_reachable_time = 86400
•   net.ipv4.neigh.default.gc_stale_time = 86400
•   net.ipv6.neigh.default.gc_thresh1 = 4096
•   net.ipv6.neigh.default.gc_thresh2 = 8192
•   net.ipv6.neigh.default.gc_thresh3 = 8192
•   net.ipv6.neigh.default.base_reachable_time_ms = 86400
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
• root file-system options: barrier=0,noatime
• datastore file-system options: noatime,barrier=0,data=writeback
• The following items were set in /opt/common_scripts/ostune_misc.sh which was executed after boot up
•   net.ipv4.tcp_syn_retries=4
•   net.ipv4.tcp_retries2=7
•   net.ipv4.tcp_keepalive_time=120
•   net.ipv4.tcp_tw_reuse=1
• httpd.conf changes
•   Timeout 20
•   KeepAlive On
•   MaxKeepAliveRequests 0
•   KeepAliveTimeout 60
•   StartServers 10
•   MinSpareServers 10
•   MaxSpareServers 10
•   ServerLimit 1536
•   MaxClients 1536
•   MaxRequestsPerChild 0
•   CustomLog logs/access_log common
• NFS export options:
•   rw,async,no_subtree_check,no_root_squash
• xnap module option used: rx_irq_mod_usec=4000

Idle Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
•   abrtd, atd, auditd, autofs, cpuspeed, cups, dovecot, httpd, ip6tables, iptables, irqbalance, kdump, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcbind, rpcgssd, rpcidmapd
• Following items were added in /etc/sysctl.conf
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.neigh.default.gc_thresh1 = 4096
•   net.ipv4.neigh.default.gc_thresh2 = 8192
•   net.ipv4.neigh.default.gc_thresh3 = 8192
•   net.ipv4.neigh.default.base_reachable_time = 86400
•   net.ipv4.neigh.default.gc_stale_time = 86400
•   net.ipv6.neigh.default.gc_thresh1 = 4096
•   net.ipv6.neigh.default.gc_thresh2 = 8192
•   net.ipv6.neigh.default.gc_thresh3 = 8192
•   net.ipv6.neigh.default.base_reachable_time_ms = 86400
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
• root file-system options: barrier=0
• The following items were set in /opt/common_scripts/ostune_misc.sh which was executed after boot up
•   vm.swappiness=0
•   vm.overcommit_memory=1

Client Driver Notes

• All prime clients were VMs running on Red Hat Enterprise Linux Server 6.1 KVM.
• The prime controller runs on the first vClient.
• Each vClient has 2 vCPUs, 4GB memory, 1 virtio_blk device and 1 virtio_net device.
• All physical clients have 2 x 10 Gbps ports, one of the 2 ports is connected to the private network,
• and is configured as a software bridge, all the vClients use this bridge.
• 7 physical clients host 28 virtual clients, 4 vClients per physical system,
• 4 physical clients host 20 virtual clients, 5 vClients per physical system.

Other Notes

• SAN Switch used: HP StorageWorks 8/24 SAN Switch
• Network Switch used: HP ProCurve 6120XG Blade Switch