Semantic segmentation, a crucial task in computer vision, has evolved to encompass point cloud data alongside traditional two-dimensional images. In this paper, we present a novel approach to perform semantic segmentation on point cloud data, leveraging two heterogenous deep neural network streams, trained over two distinct data modalities: the first stream processes two-dimensional images, while the second stream focuses on three-dimensional cloud data. By integrating these streams, our method exploits the complementary information inherent in each data modality, enhancing the segmentation accuracy and robustness. Furthermore, we propose a stacking ensemble algorithm to consolidate the outputs from both streams, enabling the final decision-making process. Ensemble methods are mainly attractive based on their premise that they offer higher accuracy when compared against the individual classifiers making them up. In stacking, multiple classification or regression models are combined using a Meta-Classifier that is trained on the outputs of the base-level models as features to deliver the final classification results. The proposed ensemble method learns a Meta-Classifier by comprising of 2D and 3D semantic segmentation deep neural networks, the two heterogeneous base classifiers. This ensemble framework combines the strengths of individual models, effectively mitigating errors and improving overall semantic segmentation performance. Experimental results on benchmark datasets demonstrate the effectiveness of the proposed approach, achieving superior segmentation accuracy compared to state-of-the-art methods. Our method holds promise for various applications, including autonomous driving, robotics, and augmented reality, where accurate and efficient segmentation of point cloud data is essential for scene understanding and decision-making.