The Object-Based Image Analysis (OBIA) paradigm strongly relies on the concept of segmentation, i.e. partitioning of an image into regions or objects that are further analyzed (e.g., described and classified). This segmentation step is thus critical, while remaining a challenging issue since there is no (and probably will never be) perfect segmentation technique yet. Indeed, various segmentation criteria as well as input data can be exploited to derive a unique segmentation that could serve for further analysis. In order to alleviate the sensitivity of the OBIA process to the segmentation step, we consider here that a set of segmentation maps can be derived from remote sensing data. Such various segmentations can be obtained through several segmentation algorithms, multiple parameter settings, or even complementary remotely-sensed observations of a given scene (possibly coming at different spatial scales). Inspired from the boosting paradigm, we thus propose a novel framework for combining multiple segmentations maps. The combination leads to a fine-grained partition of segments (superpixels) that is built by intersecting individual input partitions. More importantly, each segment is assigned a segmentation confidence score that directly relates to the correlation of the different individual segmentation inputs regarding this segment. Furthermore, each input segmentation can be assigned some local or global quality score based on expert assessment or automatic analysis. These scores are then taken into account while computing the confidence map resulting from the combination of segmentations process. It helps the process not being too much affected by incorrect segmentation inputs either at the local scale of a region or at the global scale of a map. Some experiments conducted on the ISPRS 2D Semantic Labeling dataset (made of VHR color/multispectral optical image and DSM) have been conducted to assess the relevance of the proposed framework. Results show that confidence map is a valuable information that can be produced when combining segmentations. Furthermore, the segmentation map is used to perform object-based supervised classification based on random forests. To do so, we consider several strategies in order to exploit the confidence map, either through direct thresholding or as an intermediate image on which a final segmentation is applied. A comparison with individual segmentations shows our framework is able to effectively combine various segmentation maps in order to improve the subsequent classification. From a user perspective, this approach is also able to provide a confidence map related to the geometry of the classified objects, in addition to the probability-based uncertainty map related to their semantics.