Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Power Profile set to Maximum Performance
• Thermal Configuration set to Maximum Cooling
• Turbo Mode:Enable

Storage Notes

• 2 x 600GB internal SAS drivers were connected to the LSI3008 controller,built as RAID1 and used for OS
• The SUT was connected to 1 x HUAWEI Dorado5100 via 2 x Dual port QLogic Corp ISP2532-based 8Gbps Frbre Channel HBA.
• 1 x HUAWEI Dorado5100 holds 4 x 24 x 200 GB SSD drives. There are 4 RAID10 arrays, each consisting of 24 drives . The previous two arrays each has 7 LUNs,the other two arrays each has 6 LUNs, and each LUN was partitioned for the VMs of 1 tile.

Virtualization Software Notes

• The following lines were added to /etc/sysctl.conf:
• net.ipv6.conf.all.disable_ipv6 = 1
• net.ipv4.icmp_echo_ignore_broadcasts = 1
• net.ipv4.conf.all.rp_filter = 1
• fs.inotify.max_user_watches = 65536
• net.ipv4.conf.default.promote_secondaries = 1
• net.ipv4.conf.all.promote_secondaries = 1
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.netdev_max_backlog = 500000
• fs.aio-max-nr = 70656
• vm.swappiness = 0
• vm.dirty_ratio = 10
• vm.dirty_background_ratio = 5
• kernel.sched_min_granularity_ns = 80000000
• kernel.sched_wakeup_granularity_ns = 150000000
• kernel.sched_latency_ns = 160000000
• kernel.sched_migration_cost_ns = 500000
• The following boot options were used:
• hpet=disable clocksource=tsc tsc=reliable nohz=on highres=off elevator=deadline selinux=0 audit=0 intel_iommu=on iommu=pt cgroup_disable=memory,cpu,cpuacct,blkio,hugetlb nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce nmi_watchdog=0 pci=realloc
• The following services are disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, dnsmasq, ebtables, ip6tables, iptables, irqbalance, iscsi, iscsid, ksm, ksmtuned, mdmonitor, netconsole, ntpdate, numad, oddjobd, postfix, psacct, quota_nld, radvd, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcsvcgssd, saslauthd, smartd, sssd, vncserver, winbind, ypbind
• ixgbe module option used: max_vfs=63 63 63 63 63 63 63 63 63 63 63 63 InterruptThrottleRate=4000
• 2 Intel X540 for internal communication between web server & infra server, 4 Intel 82599 configured for SUT-client and app-db communication. A maximum of 768 virtual functions were configured and 214 virtual functions were used for the web, app, db, mail infra and batch VMs.
• VMs are bound to host CPUs via their XML configuration in the following manner:
• All VMs of tiles 13-24(except infraserver24 and webserver24),dbserver25,batchserver25,appserver25,appserver26 and mailserver25 are bound to NUMA node 0
• All VMs of tiles 1-12,tile26(except appserver26),infraserver24,webserver24,infraserver25 and webserver25 are bound to NUMA node 1
• Web, Mail, and Infra VMs have a cputune shares value of 1024
• DB and APP VMs have a cputune shares value of 16384
• Batch VMs have a cputune shares value of 258
• VM configurations are as follows:
• All VMs are managed with libvirt and created using virsh define & virsh start
• All VMs are created with the following xml template, exceptions noted below template. (Each VM's xml template is at System_Under_Test/vmconfig directory):
• <domain type='kvm'>
• <name>$vm_name</name>
• <memory>$vm_memory</memory>
• <currentMemory>$vm_memory</currentMemory>
• <vcpu cpuset='$host-cpu-list'>$vm_cpu</vcpu>
• <cpu mode='custom' match='exact'>
• <model fallback='allow'>SandyBridge</model>
• <vendor>Intel</vendor>
• <topology sockets=$sockets cores=$cores threads=$threads/>
• </cpu>
• <cputune> <shares>$shares</shares> </cputune>
• <numatune>
• <memory mode='strict' nodeset='$vm_node'/>
• </numatune>
• <os>
• <type arch='x86_64' machine='pc-i440fx-1.7'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <apic eoi='on'/>
• <pae/>
• <acpi/>
• </features>
• <clock offset='utc'>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>destroy</on_crash>
• <devices>
• <emulator>/usr/libexec/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk' />
• <target dev='vda' bus='virtio'/>
• </disk>
• <hostdev mode='subsystem' type='pci' managed='no'>
• <source>
• <address domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• </hostdev>
• <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
• <listen type='address' address='0.0.0.0'/>
• </graphics>
• <memballoon model='none'>
• <alias name='balloon0'/>
• </memballoon>
• </devices>
• </domain>
• Web and Batch VMs do not use qemu for disk emulator, and therefore have the following declaration instead of 'disk type' block above:
• <controller type='scsi' model='vhost-scsi' index='0'>
• <source wwpn='naa.$vm_scsi' event_idx='on'/>
• </controller>
• The following section is repeated for the second PCI device used in the app,db,web and infra VMs:
• <hostdev mode='subsystem' type='pci' managed='no'>
• <source>
• <address domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• </hostdev>
• The following section is repeated for the second disk used in the db,mail and infra VMs for their data:
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk' />
• <target dev='vdb' bus='virtio'/>
• </disk>

Web Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• boot file-system options: barrier=0,noatime
• root file-system options: barrier=0,noatime
• nfs file-system defaults options: noatime,actimeo=14400,nocto
• MTU=9000 for eth1 (internal network).
• The MTU is set after all VMs are started through the following command: ifconfig eth1 mtu 9000 (the script setmtu.sh runs this command and the script is at System_Under_Test directory).
• Contents of nginx configuration file /etc/nginx/nginx.conf
• user nginx;
• worker_processes 4;
• worker_cpu_affinity 00000001 00000001 00000010 00000010;
• worker_rlimit_nofile 65535;
• pid /var/run/nginx.pid;
• events {
• use epoll;
• worker_connections 65535;
• multi_accept on;
• accept_mutex off;
• }
• http {
• include /etc/nginx/mime.types;
• default_type application/octet-stream;
• log_format common '$remote_addr - $remote_user [$time_local] $request"$status" $body_bytes_sent ';
• access_log /var/log/access.log common gzip buffer=4m flush=1m;
• sendfile on;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• open_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• include /etc/nginx/conf.d/*.conf;
• }
• Contents of nginx configuration file /etc/nginx/conf.d/default.conf
• server {
• listen 80 backlog=8192;
• server_name localhost;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• error_page 500 502 503 504 /50x.html;
• location = /50x.html {
• root /usr/share/nginx/html;
• }
• location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ \.php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Contents of nginx configuration file /etc/nginx/conf.d/example_ssl.conf
• server {
• listen 443 ssl;
• server_name localhost;
• ssl on;
• ssl_certificate /etc/pki/tls/certs/localhost.crt;
• ssl_certificate_key /etc/pki/tls/private/localhost.key;
• ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
• ssl_session_cache shared:SSL:1m;
• ssl_session_timeout 5m;
• ssl_ciphers ALL:!kEDH!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
• ssl_prefer_server_ciphers on;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ \.php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness = 0
• net.ipv4.tcp_fin_timeout=60
• net.ipv4.tcp_tw_recycle=1
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400
• The following lines were changed in /etc/php-fpm.d/www.conf
• listen = /dev/shm/php-cgi.sock
• listen.owner = nginx
• listen.group = nginx
• user = nginx
• group = nginx
• pm.max_children = 70

Application Server VM Notes

• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• /etc/fstab changes:
• root file-system options: barrier=0
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• * - nofile 1048576
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• Update the JVM from 1.6.0 which is installed with the Websphere 8.5.5 to 1.7.0 which is download from IBM WebSite

Mail Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline
• The following services are disabled:
• abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd svnserve winbind ypbind
• /etc/fstab changes:
• root file-system options: barrier=0
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• Dovecot was compiled with default options and the following dovecot.conf was used:
• base_dir = /var/run/dovecot/
• ssl = no
• disable_plaintext_auth = no
• listen = *
• log_path = /var/log/dovecot
• mail_debug = no
• mmap_disable = yes
• mail_fsync = never
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mdbox_rotate_size = 8192
• passdb {
• args = /etc/passwd.dovecot
• driver = passwd-file
• }
• protocols = imap
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• }
• userdb {
• driver = passwd
• }
• Following items were added to /etc/sysctl.conf:
• net.ipv6.conf.all.disable_ipv6=1
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000
• The following items were added to /etc/bashrc:
• ulimit -n 100000
• ulimit -u 10240
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400
• mysql hard memlock unlimited
• mysql soft memlock unlimited

Database Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline
• The following services are disabled:
• abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd svnserve winbind ypbind
• /etc/fstab changes:
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• root file-system defaults options:barrier=0
• Database tuning options:
• db2 update db cfg for specdb using DATABASE_MEMORY 610000
• db2set DB2_LARGE_PAGE_MEM=DB; db2set DB2_PINNED_BP=ON
• alter bufferpool IBMDEFAULTBP size 5500000
• alter tablespace roll_space NO FILE SYSTEM CACHING
• alter tablespace C_cust_space NO FILE SYSTEM CACHING
• alter tablespace C_supp_space NO FILE SYSTEM CACHING
• alter tablespace C_site_space NO FILE SYSTEM CACHING
• alter tablespace C_parts_space NO FILE SYSTEM CACHING
• alter tablespace C_custinv_space NO FILE SYSTEM CACHING
• alter tablespace M_parts_space NO FILE SYSTEM CACHING
• alter tablespace M_bom_space NO FILE SYSTEM CACHING
• alter tablespace M_wo_space NO FILE SYSTEM CACHING
• alter tablespace M_lo_space NO FILE SYSTEM CACHING
• alter tablespace M_inv_space NO FILE SYSTEM CACHING
• alter tablespace O_cust_space NO FILE SYSTEM CACHING
• alter tablespace O_ords_space NO FILE SYSTEM CACHING
• alter tablespace O_ordl_space NO FILE SYSTEM CACHING
• alter tablespace O_item_space NO FILE SYSTEM CACHING
• alter tablespace S_comp_space NO FILE SYSTEM CACHING
• alter tablespace S_sc_space NO FILE SYSTEM CACHING
• alter tablespace S_po_space NO FILE SYSTEM CACHING
• alter tablespace S_site_space NO FILE SYSTEM CACHING
• alter tablespace S_supp_space NO FILE SYSTEM CACHING
• alter tablespace U_space NO FILE SYSTEM CACHING
• db2 update database config for specdb using LOGFILSIZ 65536
• db2 update db cfg for specdb using logprimary 54 logsecond 4
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• kernel.sem = 500 64000 400 512
• kernel.shmmax = 137438953472
• kernel.shmmni = 11118096
• kernel.shmall = 8294967296
• vm.nr_hugepages=12288
• vm.hugetlb_shm_group=501
• vm.nr_overcommit_hugepages=512
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• db2inst1 soft data -1
• db2inst1 hard data -1
• db2inst1 soft nofile 65536
• db2inst1 hard nofile 65536
• db2inst1 soft fsize -1
• db2inst1 hard fsize -1
• db2inst1 soft nproc -1
• db2inst1 hard nproc -1
• * - nofile 1048576

Infraserver VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• boot file-system options: barrier=0,noatime
• root file-system options: barrier=0,noatime
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• MTU=9000 for eth1 (internal network).
• The MTU is set after all VMs are started through the following command: ifconfig eth1 mtu 9000 (the script setmtu.sh runs this command and the script is at System_Under_Test directory).
• httpd.conf changes:
• Timeout 20
• KeepAlive on
• MaxKeepAliveRequests 0
• KeepAliveTimeout 60
• StartServers 20
• MinSpareServers 20
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• StartServers 20
• MaxClients 4096
• MaxRequestsPerChild 640000
• Listen 81
• ServerName besim:80
• httpd.conf adds:
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
• FastCgiIpcDir /etc/httpd/fastcgi
• <Directory "/var/www/fcgi-bin">
• AllowOverride None
• Options +ExecCGI -Includes
• SetHandler fastcgi-script
• Order allow,deny
• Allow from all
• </Directory>
• Addhandler fastcgi-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 500
• The following httpd modules were loaded:
• file_cache mem_cache fastcgi
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness = 0
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400
• NFS export options:
• rw,no_subtree_check,crossmnt,no_root_squash

Batch Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• boot file-system options: barrier=0
• root file-system options: barrier=0
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Client Driver Notes

• The client systems VMs using Red Hat Enterprise Linux 6.4.
• One virtual client were used for each tile.
• One virtual client per tile ran all the workloads.
• Each prime client VM has 10 VCPUs, 20480 MB memory, and a 20 GB disk,1 ixgbevf device and 1 virtio_blk device are used.

Other Notes

• A Quidway S9303 36-port switch was used to connect client systems to SUT. The 10GbE ports were attached to a dual port 10 GbE SFP+ expansion module.

Performance Details: