mflops and Intel performance counters

Intel CPUs have hardware event counters which can be used to count a diverse range of events whilst a program runs. As someone interested in the performance of numerical code, I find these very useful for quickly reporting the number of MFLOPS which a program has achieved whilst running. If the number is a good percentage of the CPU's theoretical maximum performance, then there is little scope for further tuning, and if not, there may be scope.

These counters can also be used to determine the proportion of operations which execute on vector lengths of two, four, and, where relevant, eight.

This page restricts consideration to double precision arithmetic, and Linux.

Hardware Counters

Intel has long provided hardware counters, which count with zero overhead (reading them does incur a slight overhead). They are documented in chapter 19, volume 3B, of Intel's Software Developer's Manual. Unlike almost all other aspects of Intel's series of x86 processors, there is no guaranteed compatibility between different generations of CPU. Each CPU offers a different selection of events one can count, with different numeric codes to describe them.

The Linux kernel has supported reading these counters via the perf_event_open for a long while (it was introduced in the 2.6 series, and updated in the 3.x series). However, glibc provides no interface, which puts some off. Here we show some demonstration code which uses the counters, and link to other projects too.

The Linux interface is documented by man perf_event_open. This returns a file descriptor, which can be read like any other file.

The hardware usually provides four counters, and the counters are generally 48 bits wide. The code presented here makes no attempt to check for overflow. A counter incrementing at 4GHz will overflow after about 18 hours.

The mflops code

This code counts events described by Intel as:
FP_ARITH_INST_RETIRED.SCALAR_DOUBLE
FP_ARITH_INST_RETIRED.128B_PACKED_DOUBLE
FP_ARITH_INST_RETIRED.256B_PACKED_DOUBLE
FP_ARITH_INST_RETIRED.512B_PACKED_DOUBLE
These show the number of double precision instructions executed with vector lengths of one, two, four and eight respectively. The total number of floating point operations is then trivially calculated, and, if the time is known, the MFLOPS achieved too. It is also trivial to calculate the percentage of scalar operation which occured at the various vector lengths. Intel makes life easy, in that a fused multiply-add instruction adds two to the relevant counter.

$ mflops dgesv5k
  Linpack 5000x5000

      factor     solve      total     mflops       unit      ratio
  1.580E+00  0.000E+00  1.580E+00  5.277E+04  3.790E-05  2.821E+01

Time:             2.13905s
Total FP ops:   8.4182e+10
MFLOPS:            39354.8
Vector length  1:     0.51%
Vector length  2:     0.12%
Vector length  4:     0.03%
Vector length  8:    99.34%
Av vector length:     7.70

This shows the Linpack benchmark being run, and the last eight lines are produced by the mflops utility. The benchmark reports 52.77 GFLOPS, but if one includes the initialisation overheads this drops to the 39.35 GFLOPS reported by this utility.

The vectorisation report means that 99.34% of scalar operations occurred with a vector length of eight. The percentage of instructions which had a vector length of eight is only about 95.6%. Thus the average vector length of an instruction is 7.7.

But one can immediately conclude that this code is well-vectorised, and achieving a good proportion of the peak theoretical performance of the CPU (a 2.1GHz Xeon Gold with a theoretical peak of around 65 GFLOPS).

The mflops code can also report raw counter values (with the -v flag), and can count events related to IPL rather than MFLOPS (with the -ipl flag). What it cannot do is count MFLOPS or vectorisation on Haswell or Broadwell CPUs -- they lack relevant events. It works on Core 2, Nehalem, Sandy Bridge, Ivy Bridge, Skylake and Kaby Lake. It does not (currently) support AMD CPUs.

Problems?

This code relies on being able to read the performance counter events without being privileged. Some modern distributions do no permit this by default (Ubuntu 16.04LTS, on which it was developed, does). If this is a problem

    # echo 1 > /proc/sys/kernel/perf_event_paranoid
  

(as root) will fix, at, presumably, some risk to security.

The results for threaded code may not be sensible. If the code is OpenMP, setting OMP_NUM_THREADS to 1 may help.

Other codes

Other projects using the kernel's interface to the hardware performance counters:

Download mflops

mflops.c and its man page mflops.1.