Intelligent Geometric Processing and Object Information Extraction from Super Large Scale Multi-source Satellite Imagery

2021.01 - 2025.12

NSFC - State Key Program 42030102

Satellite remote sensing images have the advantages of near real-time, wide coverage and rich information. However, the production and service technology of remote sensing information lags far behind the ability of data acquisition. The joint processing and application of multi-source remote sensing satellite images is the core scientific problem of earth observation system and multi-source data fusion. Considering the scientific hot spots in the field of remote sensing earth observation and the urgent needs of national major projects, an innovative theory and method on joint intelligent processing and object information extraction from super large scale multi-source remote sensing satellite images is put forward in this proposal, based on our solid research experiences of multi-feature integrated processing of multi-source remote sensing images for over 20 years. Focuses will be concentrated on theoretical exploration and algorithm investigation of related key scientific issues, such as high reliability matching of multimodal remote sensing images, integrated on-orbit geometric calibration of multi-payload, hybrid bundle block adjustment, multi-temporal image synthesis, deep learning based intelligent extraction of ground object information, and uncertainty analysis of the whole processing chain. Our research achievements will form a significant paradigm of theoretical framework, joint processing, information extraction, and uncertainty analysis in order to give full play to each satellite, each band and each effective pixel. Moreover, our eventual aim is to break through the new round of technological blockade in the field of remote sensing from the United States, and to provide world-class technical support for the construction of global geographic information resources, the implementation of the Belt and Road initiative, emergency response to disasters, natural resource monitoring and other related projects.

Structural Reconstruction of Building Models by Integration of LiDAR Point Cloud and Multi-view Aerial Images

2019.01 - 2022.12

NSFC 41871368

3D reconstruction of building models by integration of LiDAR point clouds and aerial images has great advantages against the traditional methods that only use single data source. This proposal put forward a novel structural 3D building model reconstruction theory and the corresponding realization methods by integration of LiDAR point clouds and multi-view aerial images. Planar feature can be easily extracted from LiDAR point clouds, and linear feature can be effectively extracted from aerial images. A completely new idea of wire frame and TIN hybrid representation of 3D building model is proposed by taking the full advantages of mutual constraint mechanism of the extracted planar and linear features. The topology relationship among building vertices is also explicitly reconstructed. Several key technologies, such as 3D mesh reconstruction of point clouds with linear and planar features, building façade restoration based on image texture recognition, two-steps simplification of building models by maintaining optimum texture mapping, and structural building model reconstruction by hybrid wire frame and TIN representation, will be deeply studied. Prototype software for structural 3D building model reconstruction by integration of LiDAR point clouds and multi-view aerial images will be developed and tested with real datasets to verify the correctness and effectiveness of the proposed theory. Our achievements provide strong technical solutions for universal building vector model reconstruction, and make good foundation on both theory and applications in the fields of intelligent building, smart city, and 3D city database.

Knowledge Modelling and Public Service of Remote Sensing Imagery based on Artificial Intelligence

2018.05 - 2022.04

National Key Research and Development Program of China 2018YFB0505003

This project mainly focuses on the research of domain knowledge modeling, depth sharing and service acceleration technology to solve the problems of high-trusted knowledge modeling, demand-driven knowledge services and efficient depth sharing services in the field of global integrated observation results. The ontology knowledge base of land, ocean, surveying and mapping, meteorology and environment will be designed and established, and the decision-making knowledge depth sharing mode will be constructed to provide strong decision support for the major strategic application.

Theory and Methods on Single Stereo Model based Multi-view Imagery Dense Matching

2016.01 - 2019.12

NSFC 41571434

Dense matching with stereo images is one of the hot topics of Photogrammetry and Computer Vision communities in recent years. The key technologies are the theory and method of fast and robust pixel-by-pixel image matching. This proposal puts forward a novel dense matching theory and the corresponding realization methods of single stereo based on cost matrix and multiple soft constraints, which totally changes the traditional idea of applying constraint conditions during dense matching. A new idea of automatic optimal stereo pair selection is proposed considering multiple conditions, such as baseline length, normal vector of image plane, intersection angle, and corresponding points of aerial triangulation. We also put forward a new method of multi-view point clouds fusion in the object space by minimizing the global energy function. Line buffer and winner-takes-all strategies are adopted to get complete identical edges between image and depth map of linear features. Image segmentation results are also integrated to eliminate mis-matches in textureless areas for further improving the accuracy of matched point clouds. Finally, we will establish the new multi-view dense image matching theory and practical methods that can generate accurate point clouds with space, aerial, and low altitude images. The achievements are promising to make good foundation on both theory and applications of digital surface model and true orthoimage generation, three-dimensional building model reconstruction, and to provide strong technical supports for geographical conditions monitoring and intellectual city applications.

Digital Photogrammetry and Remote Sensing

2014.01 - 2016.12

NSFC 41322010

Limited by hardware technology and manufacturing cost, the geometrical positioning accuracy of Chinese high-resolution remote sensing satellites is far behind the advanced international level. Joint processing of multiple satellite data stands out as an optimal approach for providing a high level of positioning accuracy by efficiently utilizing the existing multi-sources geospatial information.

This project aims to research the following subjects: 1) A satellite image matching method based on region segmentation constraint. 2) A brand-new geometric calibration method of satellite three-line array imagery based on public geographic information and multi-strips dataset. 3) The joint reverse positioning and orientation of domestic remote sensing satellites under sparse and multi-sources ground control information. 4) Bundle block adjustment in large area with the aid of elevation data (SRTM). 5) An image fusion method combining both cloud extraction and high-pass filtering. 6) An automatic dodging and uniform color method for large-area remote sensing images based on standard color database.

All the above mentioned research achievements are applied in our developed satellite image processing system. It can automatically generate geo-information products with Chinese remote sensing satellite images. More than 8000 images of GF-1 and ZY-3 satellites covering the whole mainland of China have been processed with our system. At the same time, ortho maps of GF-1 wide field of view images covering China updates monthly, which greatly improves the application potentials of domestic remote sensing satellite images.

Refinement of LiDAR Point Clouds based on Combined Adjustment with Multi-sources Data

2012.01 - 2015.12

NSFC 41171292

Integrated photogrammetric processing of airborne LiDAR data and aerial images is one of the hot topics in recent years. High accuracy registration is the core technology of integrated processing of the two kinds of datasets. The rigorous geometric model of LiDAR system is analyzed, which helps to recalculate the original observations from the point coordinates. Geometric sizes of LiDAR footprints are quantitatively analyzed. Research attentions have also been paid on approximate correction of length distortions in map projection frame and direct georeferencing in national coordinates. Multi-view matching of aerial images, automatic extraction and matching of building outlines are realized by combining LiDAR point clouds and POS data. Combined bundle block adjustment with urban aerial images, POS observations, corners and feature lines extracted from LiDAR point clouds, and ground control points is performed to eliminate the systematic errors of airborne LiDAR system and thus refine the accuracy of point clouds. An improved semi-global matching (SGM) method by using mutual information as the matching cost function is put forward based on bi-lateral filtering and quad tree segmentation. Multi-dimensional dynamic programming strategy is used to minimize the global energy function, and thus get the optimized solution of dense matching. High accuracy digital surface model generation with SGM from aerial images under the assistance of LiDAR point clouds is also investigated. The advantages of aerial image and LiDAR point clouds are combined together to generate more precise 3D point clouds. Finally, the general theories and practical methods of integrated processing of LiDAR points and aerial images are built to perform the combined block adjustment and registration of the two datasets, which makes a well foundation for high accuracy digital surface model generation, true orthophoto generation and the applications of domestic LiDAR system.

Intelligent Processing System of Low Altitude UAV Imagery

2011.01 - 2013.06

National Key Research and Development Program of China 2011BAH12B05

Several key techniques are proposed in this project including automatic route recovery without any initial information, the intelligent matching of large-angle unconventional low-altitude remote sensing images and the joint bundle adjustment, automatic extraction of 3D tarrain information of low-altitude remote sensing images and the automatic color uniform and mosaic of large-scale orthophoto images. Multi-feature is used particularly to solve the problem of low accuracy and poor stability of triangulation under high reliability matching and difficult conditions. Mobile processing platforms are constructed and corresponding software systems are developed.

Aerial Triangulation with Multiple Sources of Optical Linear and Frame Images

2011.01 - 2013.12

NSFC 41071233

Integrated processing of multi-sensor optical images is one of the most recent trends in photogrammetry. The required coordinate system and image-ground transformation model for combined processing of multi-source data were established. The geocentric Cartesian coordinate system was selected to conduct the combined bundle block adjustment of line array and frame array images from different sensors. All multi-source image data with common overlapping area were introduced into a unified photogrammetric processing architecture. A high-reliability automatic matching method with multi-source and multi-view images aided by the global DEM data was proposed. The image-to-ortho matching to obtain control points and multi-view matching to obtain conjugate points were performed by adopting a two-stage parallel processing mechanism in multi-node and multi-core architectures. The theoretical and engineering methods of aero-triangulation with huge-quantity, multi-source optical images were established. The main research topics include: mathematical model of combined aero-triangulation with multi-source images, automatic weighting of different types of observations, theoretical accuracy of bundle block adjustment, and the parallel forming of large scale normal matrices. Both simulated and real project data were used to verify the theoretical analysis, and very promising results were obtained. Rigorous imaging model of linear sensors of domestic satellite was established. A new on-orbit calibration method of onboard camera based on the medium accuracy control data, which can yield accurate calibration results, was firstly proposed. The multi-strip bundle block adjustment of large areas by using the rigorous imaging model was realized. The bundle block adjustment result with the plane accuracy of 8m and height accuracy of 5m can be achieved in the absence of any ground control data. This research provides the technical support for high-accuracy, high-reliability, and rapid production of spatial information product such as DEM and DOM under complex conditions, expecting to solve the accuracy problem of global mapping by domestic remote sensing satellites.

Automatic Matching of Remotely Sensed Imagery

2007.01 - 2009.12

The National High Technology Research and Development Program of China (863 Program) 2006AA12Z136

Cutting-edge research results are achieved in this project in the fiels of fundamental theory and algorithms of image matching, automatic matching of aerial and space remote sensing imagery, high accuracy bundle adjustment, and automatic matching between remote sensing image and DOM or existing vector data. Among the above techniques, the emergency rescue and automatic matching of low-altitude images are all at the internation level. Combined with the state-of-the-art matching algorithms, the precision of the dense matching for urban scenes are increased substantially. The parallelization processing makes the efficiency of the matching algorithm 8 times higher than the traditional methods. This project provides effective technical methods for the rapid processing of massive remote sensing images, which can be applied in the fields of emergency response, land resources survay, surveying and mapping, etc.

Model of Generalized Point Photogrammetry and Application Potentials

2007.01 - 2009.12

NSFC 40671157

In the past years, the theory of Photogrammetry has advanced from point photogrammetry to line photogrammetry and hybrid point-line photogrammetry. It can be widely used for exterior orientation and 3D information extraction based on physical points and lines. However, many natural and artificial objects can not be expressed by physical points or lines, and thus can not be effectively processed by current digital photogrammetric workstations. These complex features put forward challenges to the current theory and methods.

This project extends the concept of physical point to mathematical generalized point, and constructs the new uniform model of generalized point photogrammetry based on traditional collinearity equations. The biggest advantage of generalized point photogrammetry is that the exact conjugacy between ground point and image point of a feature is not essential, the whole feature correspondency between them is enough for generalized point photogrammetry, which is an evolution of the photogrammetric theory. To establish an uniform model of generalized point photogrammetry and corresponding algorithms, models of relative orientation, intersection, resection, bundle adjustment and error propagation with multiple features are thoroughly studied. Multiple features based multi-view image matching and block adjustment of images acquired with linear and frame sensors are also the innovation of this project. Model-driven 3D building modeling based on generalized point photogrammetry and museum pieces 3D reconstruction based on digital images and laser point clouds are also realized. The achievements of this project have excellent potential in image matching, digiral ortho image generation, digital city modeling, stereo mapping and 3D terrain information extraction areas.

Three Dimensional City Modelling Techniques with Videogrammetry

2004.01 - 2006.12

NSFC 40301041

Three-dimensional (3D) city model is one of the most important components of digital city and also digital earth. It has a promising future in Car Navigation, Tourism and Marketing, Architecture, City Planning, Environment Protection and so on, which attracts much attention from related communities and governments all over the world. At present, due to the imperfection of theory and technology, 3D city models are obtained by purely manpower, which is time-consuming with high cost. How to reduce the cost and improve the efficiency of 3D city model generation is the key problem that needs to be resolved urgently. This project is based on the theories of Digital Photogrammetry and Computer Vision. Non-metric digital video camera is used to obtain image sequences (aerial and terrestrial) with high overlapping and short-baseline. Coarse 2D vector data of buildings and LiDAR data are also used as sources of information. Image sequences based Videogrammetry is used to reconstruct 3D city model. Initial values of camera parameters of each image are given by automatic aerial triangulation technique for high overlapping image sequences. The correspondence between space edge of building and its line feature in image sequences is determined by automatic image matching and multi-view geometry. Hybrid point-line photogrammetry with additional constraints is used to reconstruct buildings accurately. The final achievements of this project are the theory and software, which can automatically generate 3D city model with fine texture, update low precision digital map, extract DEM and produce orthoimages. Experiments show that the efficiency of the developed system is eight times higher than that of purely manpower. The research project not only reduces the costs and improve efficiency of 3D city model generation, but also enriches and evolves the theory of Digital Photogrammetry toward fully automatic. Results of the project will also make a well foundation of various applications such as 3D city model generation and creation of digital city.