BenchSpy - Standard Report

StandardReport supports three types of data sources:

• Direct
• Loki
• Prometheus

Each allows you to use either pre-defined metrics or create custom ones.

Pre-defined (Standard) Metrics

Direct and Loki

Both query executors focus on the characteristics of the load generated by WASP. Their datasets are nearly identical because the Direct executor queries load-specific data before it is sent to Loki. The Loki executor, however, offers richer querying options (via LogQL) and access to the actual load profile (rather than the configured one).

Predefined metrics for both include:

• Median latency
• 95th percentile latency
• Max latency
• Error rate

Latency is the round-trip time from sending a request to receiving a response from the Application Under Test (AUT).

Error rate is the ratio of failed responses to total responses. This includes requests that timed out or returned errors from Gun or Vu implementations.

Prometheus

Prometheus metrics focus on resource consumption by the AUT rather than load generation. These metrics include:

• Median CPU usage
• 95th percentile CPU usage
• Max CPU usage
• Median memory usage
• 95th percentile memory usage
• Max memory usage

Prometheus queries focus on total consumption, which combines the usage of the underlying system and your application.

How to Use Pre-defined Metrics

To use pre-defined metrics, call the NewStandardReport method:

report, err := benchspy.NewStandardReport(
    "v1.1.1",
    // Query executor types for which standard metrics should be generated
    benchspy.WithStandardQueries(
        benchspy.StandardQueryExecutor_Prometheus,
        benchspy.StandardQueryExecutor_Loki,
    ),
    // Prometheus configuration is required if using standard Prometheus metrics
    benchspy.WithPrometheusConfig(benchspy.NewPrometheusConfig("node[^0]")),
    // WASP generators
    benchspy.WithGenerators(gen),
)
require.NoError(t, err, "failed to create the report")

Custom Metrics

WASP Generator

Since WASP stores AUT responses in each generator, you can create custom metrics based on this data. For example, to calculate the timeout ratio:

var generator *wasp.Generator

var timeouts = func(responses *wasp.SliceBuffer[*wasp.Response]) (float64, error) {
    if len(responses.Data) == 0 {
        return 0, nil
    }

    timeoutCount := 0.0
    inTimeCount := 0.0
    for _, response := range responses.Data {
        if response.Timeout {
            timeoutCount++
        } else {
            inTimeCount++
        }
    }

    return timeoutCount / (timeoutCount + inTimeCount), nil
}

directExecutor, err := NewDirectQueryExecutor(generator, map[string]DirectQueryFn{
    "timeout_ratio": timeouts,
})
require.NoError(t, err, "failed to create Direct Query Executor")

Loki

Creating custom LogQL queries is even simpler. Use NewLokiQueryExecutor with a map of desired queries:

var generator *wasp.Generator

lokiQueryExecutor := benchspy.NewLokiQueryExecutor(
    map[string]string{
        "responses_over_time": fmt.Sprintf("sum(count_over_time({my_label=~\"%s\", test_data_type=~\"responses\", gen_name=~\"%s\"} [5s])) by (node_id, gen_name)", label, gen.Cfg.GenName),
    },
    generator.Cfg.LokiConfig,
)

Prometheus

Adding custom PromQL queries is equally straightforward:

promConfig := benchspy.NewPrometheusConfig()

prometheusExecutor, err := benchspy.NewPrometheusQueryExecutor(
    map[string]string{
        "cpu_rate_by_container": "rate(container_cpu_usage_seconds_total{name=~\"chainlink.*\"}[5m])[30m:1m]",
    },
    promConfig,
)
require.NoError(t, err)

Using Custom Queries with StandardReport

To use custom queries in a StandardReport, pass the custom QueryExecutors created above using the WithQueryExecutors option instead of the StandardQueryExecutorType:

report, err := benchspy.NewStandardReport(
    "v1.2.3",
    benchspy.WithQueryExecutors(directExecutor, lokiQueryExecutor, prometheusExecutor),
    benchspy.WithGenerators(gen),
)
require.NoError(t, err, "failed to create baseline report")