Proxy App Quality Standards and Best Practices

This page describes the quality standards and best practices for ECP proxy
apps. Proxies that comply with these standards will be more usable in
co-design engagements and will receive preferential treatment from the ECP
Proxy App team. Standards and practices in this guide are designated as either
required or recommended. Over time, we expect to increase the overall
quality of the proxy app suite by promoting some recommended practices to
requirements.

Code Distribution and Availability

Required: All proxy app source code must be publicly available.

Required: Provide a stable release tarball or a git tag or commit that
points to a stable release.

Recommended: A source code repository (e.g. on GitHub) is the preferred
mechanism to distribute source code as it also provides an easy way of
funneling changes from the proxy team or vendors back to the upstream
developers. Distributing code as a web link to a tarball is discouraged but
acceptable. Private git/svn repositories or email/forms to request source
code are not acceptable.

Recommended: Create a spackage to distribute your proxy through
Spack. The Proxy App Team will assist with the
creation of spackages.

Recommended: Proxy apps should be self-contained. Proxy apps with large or
complex library dependencies will be difficult or impossible to run on minimal
testbed or simulator platforms typically used by vendors during early
design phases. If dependencies are necessary, creation of a spackage to
distribute the proxy along with the required dependencies will greatly assist
potential proxy users. Note that Spack already ships many libraries (e.g.
lapack, kokkos).

Build Systems

Required: The build system should expose common flags and variables (e.g.
CXX, CXXFLAGS, CC, CFLAGS, FC, FLAGS, LDFLAGS) for easy testing.

Required: The proxy app should be able to be built on a standard Linux
distribution without need for modification of the standard FHS (Filesystem
Hierarchy Standard).

Required: The proxy app should be able to be built and adjusted without
deep changes to the build system (e.g. commenting and uncommenting lines in the
makefile is fragile to automate).

Required: Provide mechanism (e.g. searching PATH, LIBNAME_DIR) to
allow for passing of any library dependencies to the build system.

Recommended: Easy integration with Spack. This should be handled through
the above requirements.

Design

Recommended: Follow DRY (Don’t Repeat Yourself) practices and refactor code
common or beneficial to multiple applications into library dependencies.

Recommended: Keep the number of lines of code to a minimum without negating
the intended purpose of the proxy application. A larger codebase may allow for
further exploration of the design space but adds to the difficulty of working
with the proxy. A suggested goal is to have an proxy application with fewer
than 10k lines of code.

Three Expected Running Modes to be specified

Required: A quick test to verify correct compilation and, if applicable,
track kernel performance. This should have an execution time on the order of
seconds but can be larger if necessary. See Verification of Correctness below.

Recommended: A running mode (weak and strong scaling) representing a current workload.

Recommended: A running mode (weak and strong scaling) representing an exascale workload.

General Documentation

General Description of Proxy Application

Required: A general description of what the proxy is and which algorithms
it comprises/computational science motifs it represents.

Required: Proxy apps should have documentation of the general scope of the
proxy app and where it does/doesn’t represent the parent application correctly.

Intent and Limitations of Proxy Application with Respect to Parent

Recommended: Documented analysis of how the app proxy represents the parent
application.

Recommended: Documented aspects of the proxy app that may look simple
enough that some quick optimizations can be applied but should not be
applied because they will not be feasible in the parent application.

Required: Documentation should also mention what aspect of the parent
application the proxy is a proxy for (e.g. computation, memory, communication,
I/O).

Building and running/testing

Required: Documentation should cover the build process (including
dependencies) and how to test the final executable. See Verification of
Correctness below.

Required: Documentation should cover all configuration options and
command-line parameters a user is expected to use.

Recommended: Documentation should explain why the user might use each
option, not just what each option enables.

Running mode documentation

Required: The running mode documentation should not only include input
decks but should also include expected outputs for each of the input decks. See
Verification of Correctness below.

Recommended: The documentation should discuss minimum/maximum
reasonable problem sizes to prevent users running at extreme scaling points or
running unrealistic cases.

Verification of Correctness

Required: Inclusion of an automatable regression test that checks the final
output against a known good reference.

Recommended: Regression test that compares final output against a reference
within an acceptable threshold based upon the science being simulated.

Recommended: Tests should cover most code paths provided by the proxy app.
/the Proxy App team can help with measuring code coverage numbers (e.g.,
reported performance metrics/models).

Reported Performance metrics/models

Required: The proxy app should include a figure of merit (FOM) that vendors
can use to compare performance under different conditions. This could be based
on the science being simulated or what the proxy app is stressing (e.g.,
computational power, memory etc.). The FOM should be representative of the
performance of the parent application

Recommended: If applicable, include one or more timers around the most
important code regions and an option to output the execution time of those
regions.

Recommended: If applicable, include a performance model based on the
resource the proxy app is stressing.

ECP Proxy Applications Website Contributions

Required: Provide a description of the proxy app.

Recommended: Contribute tags to facilitate indexing of proxy apps.

You can submit your app on this site.