Edge computing is characterized by heterogeneous hardware, distributed deployment, and a need for on-site processing, which makes performance benchmarking challenging. This paper presents SHEAB (Scalable Heterogeneous Edge Automation Benchmarking), a novel framework designed to securely automate the benchmarking of Edge AI devices at scale. The proposed framework enables concurrent performance evaluation of multiple edge nodes, drastically reducing the time-to-deploy (TTD) for benchmarking tasks compared to traditional sequential methods. SHEAB’s architecture leverages containerized microservices for orchestration and result aggregation, integrated with multi-layer security (firewalls, VPN tunneling, and SSH) to ensure safe operation in untrusted network environments. We provide a detailed system design and workflow, including algorithmic pseudocode for the SHEAB process. A comprehensive comparative review of related work highlights how SHEAB advances the state-of-the-art in edge benchmarking through its combination of secure automation and scalability. We detail a real-world implementation on eleven heterogeneous edge devices, using a centralized 48-core server to coordinate benchmarks. Statistical analysis of the experimental results demonstrates a 43.74% reduction in total benchmarking time and a 1.78× speedup in benchmarking throughput using SHEAB, relative to conventional one-by-one benchmarking. We also present mathematical formulations for performance gain and discuss the implications of our results. The framework’s effectiveness is validated through the concurrent execution of standard benchmarking workloads on distributed edge nodes, with results stored in a central database for analysis. SHEAB thus represents a significant step toward efficient and reproducible Edge AI performance evaluation. Future work will extend the framework to broader workloads and further improve parallel efficiency.

The transition of Large Language Models (LLMs) from centralized clouds to edge environments is critical for addressing privacy concerns, latency bottlenecks, and operational costs. However, existing edge benchmarking frameworks remain tailored to discriminative Deep Learning tasks (e.g., object detection), failing to capture the multidimensional challenges of generative AI, specifically the trade-offs between token generation speed, semantic accuracy, and hardware sustainability. To address this gap, we introduce LEAF (LLM Edge Assessment Framework), a novel evaluation methodology that integrates Circular Economy principles directly into performance metrics. LEAF assesses edge deployments across five synergistic pillars: Circular Economy Score, Energy Efficiency (Joules/Token), Performance Speed (Tokens/Second), semantic accuracy (BERTScore), and End-to-End Latency. We validate LEAF through an extensive experimental analysis of five distinct hardware classes, ranging from embedded IoT devices (Raspberry Pi 4 and 5, NVIDIA Jetson Nano) to professional edge servers (NVIDIA T400) and repurposed legacy workstations (NVIDIA GTX 1050 Ti). Utilizing 4-bit quantized models via the Ollama runtime, our results reveal a counterintuitive insight: repurposed consumer hardware significantly outperforms modern purpose-built edge SoCs. The legacy GTX 1050 Ti achieved a 20× speedup over the Raspberry Pi 4 and maintained superior energy-per-task efficiency compared to low-power ARM architectures by minimizing active runtime. These findings challenge the prevailing narrative that newer silicon is essential for Edge AI, demonstrating that sustainable, high-performance inference can be achieved by extending the lifecycle of existing hardware. LEAF thus provides a blueprint for a “Green Edge” ecosystem that balances computational capability with environmental responsibility.

Edge computing has emerged as an important paradigm to address the evolving demands of latency-sensitive applications and the adaptation of Internet of Things (IoT) devices, offering near edge data processing to increase performance, reduce bandwidth usage, and increase data privacy. However, benchmarking these heterogeneous edge environments remains a challenge due to their distributed nature and diverse hardware configurations. This paper proposes "Scalable Heterogeneous Edge Automation Benchmarking"(SHEAB), a novel containerized and automated framework designed to evaluate edge computing systems comprehensively. SHEAB integrates containerization for portability, automation for efficiency, and a multi-layered security architecture - including firewalls, VPNs, and secure shell connections - to ensure robust data integrity across varied edge servers. This research advances the field by providing a scalable, secure, and adaptable benchmarking solution, with future directions aimed at researching hardware capability assessments and increasing AI-driven edge-computing testing and benchmarking.