Optimize the VM Performance Query with Splunk

If you have integrated with Splunk to retrieve guest OS performance data, when Commander refreshes VM performance for a machine (either manually for a single VM, or automatically with the nightly performance update), Commander generates a query for the VM and sends it to Splunk to perform. This query can take a long time to complete, especially when every VM in your environment is being queried. To reduce the time taken and reduce the load on your Splunk server when performance data is being requested, you can configure Splunk to run these queries for all the hosts in your environment and store the results for Commander to fetch later.

Make sure you run each saved search at least once in Splunk to generate the index. Otherwise, the index must be generated when Commander executes the search for the first time, resulting in a very slow performance update.

About performance sample size

Before configuring the saved searches in Splunk, you need to understand how Commander treats the performance data it retrieves. For each performance counter (for example, Percent CPU), Commander retrieves a daily sample from Splunk and detects how many days of performance data are available; the number of days Commander retrieves is known as the sample size. Commander uses both the sample size and the number of available samples to determine whether this performance data can be used to drive recommendations. For example, if a machine is only active on weekdays and you would like to make sure that a week's data is present before applying it to recommendations, configure the sample size to be 5 and the minimum number of days in the sample to be 7.

You will need the sample size when configuring the searches in Splunk. The example queries in this topic use the numbers given above: sample size = 5 days, and minimum number of samples = 7.

Configure Accelerated Search in Splunk

By making use of the Report Acceleration Summaries feature in Splunk, you can predefine a search that's run once for each performance counter that Commander needs to refresh. You must ensure that report acceleration has been enabled for each query, and that the query has been scheduled to run once a day. Since Commander summarizes the data daily, it's not necessary for Splunk to perform the query multiple times in a day; just make sure that the query runs before midnight, when the nightly performance job in Commander starts.

These instructions assume that you have already created and configured the data inputs for the target hosts.

Percent CPU Performance Counter

In Splunk, create a new report with the following settings. See Customize the Splunk Accelerated Search below if you need to specify different values for Destination App or Search Name.

Destination app

Search

Search name

vCommander-windows-cpu

Available object

PerfOS_Processor

Counters

PercentProcessorTime

Name (optional)

Search

PercentProcessorTime=* Name=_Total latest=@d earliest=@d-7d | bucket _time span=5m | stats avg(PercentProcessorTime) as avgBucketValue, min(_time) as minBucketTime, max(_time) as maxBucketTime by host, _time | bucket _time span=1d | stats avg(avgBucketValue) as AVERAGE, max(avgBucketValue) as PEAK, min(minBucketTime) as minDayTime, max(maxBucketTime) as maxDayTimeUnadjusted by host, _time | eval maxDayTime = maxDayTimeUnadjusted + 300

Change the value of the earliest parameter to match your sample size. For example, to sample a month's data, change the earliest value to: earliest=@d-30d

Accelerate this search

Enabled

Summary range

Must be greater than the sample size you've chosen

Schedule this search

Enabled

Available Bytes Performance Counter

In Splunk, create a new search with the following settings. See Customize the Splunk Accelerated Search below if you need to specify different values for Destination App or Search Name.

Destination app

Search

Search name

vCommander-windows-available-bytes

Available object

PerfOS_Memory

Counters

AvailableBytes

Search

AvailableBytes=* latest=@d earliest=@d-7d | bucket _time span=5m | stats avg(AvailableBytes) as avgBucketValue, min(_time) as minBucketTime, max(_time) as maxBucketTime by host, _time | bucket _time span=1d | stats min(avgBucketValue) as PEAK, min(minBucketTime) as minDayTime, max(maxBucketTime) as maxDayTimeUnadjusted by host, _time | eval maxDayTime = maxDayTimeUnadjusted + 300

Change the value of the earliest parameter to match your sample size. For example, to sample a month's data, change the earliest value to: earliest=@d-30d

Accelerate this search

Enabled

Summary range

Must be greater than the sample size you have chosen

Schedule this search

Enabled

Pages Output per Second Performance Counter

In Splunk, create a new search with the following settings. See Customize the Splunk Accelerated Search below if you need to specify different values for Destination App or Search Name.

Destination app

Search

Search name

vCommander-windows-pages-output-per-sec

Available object

PerfOS_Memory

Counters

PagesOutputPersec

Search

PagesOutputPersec=* latest=@d earliest=@d-7d | bucket _time span=5m | stats avg(PagesOutputPersec) as avgBucketValue, min(_time) as minBucketTime, max(_time) as maxBucketTime by host, _time | bucket _time span=1d | stats avg(avgBucketValue) as AVERAGE, max(avgBucketValue) as PEAK, min(minBucketTime) as minDayTime, max(maxBucketTime) as maxDayTimeUnadjusted by host, _time | eval maxDayTime = maxDayTimeUnadjusted + 300

Change the value of the earliest parameter to match your sample size. For example, to sample a month's data, change the earliest value to: earliest=@d-30d

Accelerate this search

Enabled

Summary range

Must be greater than the sample size you've chosen

Schedule this search

Enabled

Configure Commander to use Splunk Accelerated Search

Access:

Configuration > System > Integration tab

Available to:

Commander Role of Superuser

To configure Commander to make use of the saved searches to update performance data:

  1. On the Integration page, under Splunk Server, click Edit.
  2. In the Splunk Server dialog, enable Accelerate Searches.
  3. Click Test.

    Commander queries Splunk to determine whether everything is set up correctly.

  4. Once the test has succeeded, click OK to save the configuration.

If Splunk has been misconfigured and Accelerate Searches is enabled, Commander won't be able to retrieve performance data from Splunk until the Splunk configuration has been corrected.

Customize the Splunk Accelerated Search

Use Different Values for Destination App or Search Name

Commander expects the Splunk values for Destination App and Search Name to match those in the tables above. If you can't use the expected values (for example, if you need to point to a differently named app, or if you have a policy for search names), you can customize these values.

  1. Create a splunk.properties file in the <vcommander_install_dir>/tomcat/common/classes directory.
  2. Use the standard property file format, one property per line:

    property1=value1

    property2=value2

    ...

  3. Add the following properties:
  • splunk.properties.application — The name of the application where the searches have been created. The default value is search.
  • splunk.performance.windows.percent_cpu — The name of the saved search for reading the percent CPU performance counter. The default value is vCommander-windows-cpu.
  • splunk.performance.windows.available_bytes — The name of the saved search for reading the available bytes performance counter. The default value is vCommander-windows-available-bytes.
  • splunk.performance.windows.pages_out_per_sec — The name of the saved search for reading the pages output per second performance counter. The default value is vCommander-windows-pages-out-per-sec.

Customize the Search Query

VMware takes the average value of every counter in a five-minute sampling period and uses that value for its calculations. Performance queries used by Commander are likewise broken into five-minute intervals to match the way VMware retrieves performance data. Commander identifies the peak value and the average value for the day; these values are used to drive rightsizing recommendations.

In order for Commander to use both VM and Guest OS performance data to drive rightsizing recommendations, the two counters need to be comparable. Keep this in mind if you want to customize the search query.