Internet of Digital Reality (IoD) is a concept that extends the Internet of Things (IoT) with the management, transmission and harmonization of digital realities. IoD covers aspects of connectivity, accessibility and usability with respect to different cognitive entities present in the digital world, via a confluence of technologies including virtual reality, artificial intelligence and 2D digital environments, in a way that recognizes human factors and cognitive aspects as key issues. Devices, interfaces, and interacting entities can be enabled through IoD to share digital realities and to thereby build a new level of reality, using intelligent connections mostly based on immersive virtual scenarios and multi-modal interactions in both public and private networks. In this paper, we look into the infrastructural requirements of and challenges behind the Internet of Digital Reality, which must be solved in order to deliver a high-quality user experience while keeping the increasing complexity of these networks at bay.