EXtended Reality (XR) alongside the Digital Twin (DT) in Industrial Internet of Things (IIoT) emerges as a promising next-generation technology. Its diverse applications hod the potential to revolutionize multiple facets of Industry 4.0 and serve as a cornerstone for the rise of Industry 5.0. However, current systems are still not effective in providing a high-quality experience for users due to various factors, one of which is their limited resources for processing and transmitting complex data and big data. To overcome these challenges, this paper presents an in-depth analysis of performance optimization techniques for resource-constrained Augmented Reality (AR) and/or Virtual Reality (VR) environments operating with DT, with a specific focus on Quality of Service (QoS), Quality of Experience (QoE), Edge-Cloud architectures and future research directions. Furthermore, this study delves into the intricate complex trade-off relationships involving optimization factors, including system quality, information quality, and QoE. In addition, it also explores potential solutions based on powerful emerging technological tools, including data compression, blockchain, cloud computing, quantum computing, Artificial Intelligence (AI) / Machine Learning (ML), and cybersecurity in the Cyber-Physical Systems (CPS). The insights provided in this comprehensive survey can inspire and guide researchers and industrial practitioners in optimizing performance for XR with DT applications in resource-constrained Smart Manufacturing System (SMS).