High-Performance LINPACK (HPL) Benchmarking

HPL (High-Performance LINPACK) — the industry-standard benchmark suite behind the TOP500 list — deployed, measured, and scientifically analyzed across four cluster configurations to establish the edge platform's raw GFLOPS capacity.

1. Overview and Objectives

The objective of Task 2 is to investigate the computing performance in GFLOPS (Giga Floating-Point Operations Per Second) of the edge computing cluster. HPL measures the execution speed with which a distributed cluster solves a dense system of linear equations (Ax = b) via LU factorization with partial pivoting.

2. Cluster Specifications

3. Installation and Deployment in Air-Gapped Environment

Because the cluster operates in an isolated, air-gapped edge environment without internet access, HPL was provisioned via the central NFS network share (/shared/mpi/HPL).

Components Used

• HPL Binary (xhpl): HPL version 2.3, pre-compiled for ARM64 (aarch64) with MPI and BLAS support.
• OpenBLAS Math Library (libopenblas.so.0): Highly optimized LAPACK/BLAS routines for dense linear algebra.
• OpenMPI (5.x): Message Passing Interface for cross-node inter-process communication.

NFS Directory Layout

/shared/mpi/HPL/
├── xhpl                     # HPL benchmark executable
├── HPL.dat                  # Parameter configuration file
├── libopenblas.so.0         # Dynamic OpenBLAS library
├── libopenblasp-r0.3.29.so  # OpenBLAS shared object
└── hostfile_all             # MPI host configuration

Critical MPI Configuration Parameters

• -x LD_LIBRARY_PATH=/shared/mpi/HPL:$LD_LIBRARY_PATH — Forces all worker processes on remote Pis to dynamically link against the shared OpenBLAS library directly from the NFS mount.
• --mca btl_tcp_if_include 10.0.0.0/24 — Forces OpenMPI to route traffic exclusively across the physical cluster network segment, preventing MPI from inadvertently binding to the internal Kubernetes/Flannel overlay network (10.244.x.x).
• --bind-to none — Disables strict CPU core binding (hwloc) to eliminate scheduling conflicts across heterogeneous CPU architectures (Cortex-A76 vs. Cortex-A53).

4. Benchmark Methodology & Execution

Parameter Tuning in HPL.dat

Parameters were carefully calibrated to accommodate the strict memory ceiling of the 1 GB Raspberry Pi 3 workers:

• Problem Size (N = 4096): Total matrix memory footprint is 4096 × 4096 × 8 Bytes ≈ 134.2 MB. Partitioned across 36 processes on 9 nodes, each node holds only ~15 MB in RAM, reliably avoiding the Linux Out-Of-Memory (OOM) killer on memory-constrained workers.
• Block Size (NB = 128): Calibrated for the L1/L2 cache line size and register pipeline of ARM Cortex-A architectures.
• Process Grid (P × Q): Configured to match the available physical cores at each scaling stage (P × Q = Cores):
  • 1 Node: P = 2, Q = 2 (4 cores)
  • 2 Nodes: P = 2, Q = 4 (8 cores)
  • 4 Nodes: P = 4, Q = 4 (16 cores)
  • 9 Nodes: P = 6, Q = 6 (36 cores)

Executed Benchmark Commands

# 1 Node (Master Pi 5, 4 Cores):
mpirun -np 4 -x LD_LIBRARY_PATH=/shared/mpi/HPL ./xhpl

# 2 Nodes (Pi 5 + 1× Pi 3, 8 Cores):
mpirun --hostfile hostfile_2nodes --mca btl_tcp_if_include 10.0.0.0/24 \
       -x LD_LIBRARY_PATH=/shared/mpi/HPL:$LD_LIBRARY_PATH -np 8 ./xhpl

# 4 Nodes (Pi 5 + 3× Pi 3, 16 Cores):
mpirun --hostfile hostfile_4nodes --mca btl_tcp_if_include 10.0.0.0/24 \
       -x LD_LIBRARY_PATH=/shared/mpi/HPL:$LD_LIBRARY_PATH -np 16 ./xhpl

# 9 Nodes (Pi 5 + 8× Pi 3, 36 Cores — Full Cluster):
mpirun --hostfile hostfile_all --mca btl_tcp_if_include 10.0.0.0/24 \
       -x LD_LIBRARY_PATH=/shared/mpi/HPL:$LD_LIBRARY_PATH -np 36 ./xhpl

5. Results and Evaluation (GFLOPS)

All benchmark runs completed successfully and satisfied HPL's formal residual validation (PASSED with normalized residual < 16.0).

Scaling Behavior (Strong Scaling at N = 4096)

GFLOPS Scaling at fixed problem size N=4096 (Strong Scaling):

  GFLOPS
   3.5 ┼─── 3.018 (Pi 5 solo)
   3.0 ┤
   2.5 ┤
   2.0 ┤          1.986 (2 Nodes)
   1.5 ┤                   1.839 (4 Nodes)
   1.0 ┤                            1.652 (9 Nodes / 36 Cores)
   0.5 ┤
   0.0 ┴───────┬────────────┬────────────────┬──────────────
            4 Cores      8 Cores          16 Cores        36 Cores

6. Scientific Analysis & Bottleneck Identification

Three primary architectural factors explain the performance drop when scaling across multiple nodes:

1. Communication Overhead vs. Computation Time (Amdahl's Law):
   At N = 4096, total computational workload is approximately ⅔ N³ ≈ 4.58 × 10¹⁰ FLOPs. Distributed across 36 cores, each core computes its submatrix in mere milliseconds. Consequently, the vast majority of the 27.75 s total runtime is spent exchanging pivot vectors and matrix panels over the network rather than computing.
2. Heterogeneous Network Bottleneck (1000 Mbps vs. 100 Mbps):
   The Pi 5 features native Gigabit Ethernet, whereas the Pi 3 Model B workers only provide 100 Mbps Fast Ethernet (internally routed through a shared USB 2.0 bus controller). Network latency and bandwidth throttling on the worker links dominate total execution time the moment data crosses node boundaries.
3. Heterogeneous CPU Performance & Barrier Synchronization:
   The Cortex-A76 cores on the Pi 5 (out-of-order execution, 2.4 GHz, wide SIMD pipelines) are 4–5× faster per core than the in-order Cortex-A53 cores on the Pi 3 (1.2 GHz). In a synchronous MPI grid (P × Q), the fast master node is forced to stall at every synchronization barrier waiting for the slowest worker.

7. Conclusion & Architectural Takeaway

• Verification of Deliverables: HPL was successfully built, deployed over NFS in an air-gapped environment, executed across the full 9-node/36-core cluster, and validated with zero residual errors.
• Architectural Takeaway: For compute-intensive workloads on heterogeneous edge clusters connected by 100 Mbps Ethernet, synchronous, tightly coupled MPI decomposition is ill-suited due to barrier stalls and network bottlenecks. Asynchronous task-queue models (e.g., Celery/MQTT or independent Kubernetes pod replicas, as deployed in the monitoring stack) scale far more gracefully.