Researchers at the REEF are developing vision-based guidance, navigation, and control algorithms for micro air vehicles (MAVs).  This research is supported by the Active Vision for the Control of Agile, Autonomous Flight (AVCAAF) grant, funded jointly by AFOSR and AFRL/MN, as well as the Research Institute for Autonomous Precision Guided Systems (RIAPGS) grant, funded by AFOSR.  These projects, which involve researchers from multiple universities, support the general objective of developing technologies to enable MAVs to operate autonomously in urban environments.  Vision-based GNC entails using images collected from an onboard camera (or cameras) to estimate the motion of the vehicle as well as the locations of obstacles in the three-dimensional environment.  This information is then used for the purposes of path planning and control.  Vision-based GNC is a broad research field, and REEF researchers are involved in several different approaches, some of which are described below.

Vision-Based Geometry Estimation, Path Planning, and Control    

Collaborators:    Richard J. Prazenica     UF-REEF, Virginia Tech
                             Andrew J. Kurdila         Virginia Tech
                             Robert C. Sharpley       University of S. Carolina
                             Peter Binev                 University of S. Carolina
                             Matt Hielsberg             University of S. Carolina
                             John Lane                   University of S. Carolina
                             Johnny Evers               AFRL/MN

 This research is focused on developing vision-based control strategies that use images obtained from a single onboard camera to estimate the three-dimensional geometry of the environment (i.e., the locations of buildings and other obstacles).  This information is essential in order to plan a collision-free trajectory for a MAV operating in an urban environment.  A typical mission considered in this line of research is depicted in Figure 1 in which the objective is to successfully navigate a MAV from point A to point B in an urban scene without any a priori knowledge of the environment. 

The overall vision-based control architecture employed in this research is illustrated in Figure 2.  This control system is being developed and tested using simulations conducted in the REEF Visualization Laboratory.  A dynamic model is used to simulate the motion of the MAV (i.e., its position and orientation) in response to a set of control inputs.     The Visualization Lab runs software that generates synthetic camera images corresponding to the position and orientation of the vehicle in the virtual environment.  These images are processed using a feature point tracker, which selects points in the images (e.g., building corners) and tracks them in subsequent image frames.  Feature points that have been tracked in multiple image frames can be used to estimate the translational and rotational motion of the vehicle.  As shown in the diagram, the motion estimation can also incorporate data from any other sensors on the vehicle such as GPS, accelerometers, or gyros.  Given a set of tracked feature points and an estimate of the vehicle motion, the next step is to use epipolar geometry to compute the three-dimensional locations of the points within the environment.   The collection of three-dimensional points is a representation of the structure of the scene, but is not convenient for the purposes of planning obstacle-free trajectories.  Therefore, the three-dimensional points are processed by an adaptive learning algorithm which generates a mathematical representation of the scene in terms of piecewise-constant functions.  The algorithm adapts this representation as more data are received, employing more functions in areas where there are large variations in the data.  This mathematical representation of the scene provides obstacle avoidance constraints for the path planning and control algorithms.  Two general approaches have been taken to the path planning and control problem, both based on the concept of receding horizon control.  Receding horizon control entails computing a control history that optimizes a cost function (e.g., minimize flight time, approach a goal location…) over a relatively short time horizon.  The computed control history is then applied over a subset of this time horizon and the process is repeated.  In the first approach, a set of local path points are computed such that the vehicle moves towards the goal while avoiding obstacles in the environment.  The vehicle is then commanded to fly through a subset of these path points after which a new path is planned.  The second approach essentially combines path planning and control into a single process.  Instead of explicitly planning a trajectory, an optimal control sequence (or optimal set of maneuvers) is computed that results in a local, obstacle-free trajectory while approaching the goal.           

Figure1:  A typical MAV mission through an urban environment.

Figure 2:  Vision-based control architecture.

An example of a simulated MAV flight through the UF campus virtual environment is illustrated in Figures 3-6.  In this example, a MAV starts at point A, traveling east on University Ave., and is directed to fly to point B where a van is parked on the south side of Ben Hill Griffin Stadium.  The MAV has no a priori knowledge of the environment, and a direct flight from point A to point B would result in the MAV crashing into the side of the stadium.   The flight simulation model employed in this study, corresponding to a University of Florida MAV with a 6 in. wing span, was generated by Martin Waszak at the NASA Langley Research Center.  During the simulated flight, the vision-based control system adaptively estimates the geometry of the environment by processing synthetic images obtained from a camera pointed out the nose of the aircraft.  In this example, it is assumed that the position and orientation of the vehicle is known at all times, eliminating the need for the motion estimation process in Figure 2.  Future studies will address the more difficult problem of incorporating vision-based motion estimation into the simulation.  The receding horizon controller selects an optimal sequence of three control maneuvers, where each maneuver is chosen from a set of 25 maneuvers.  These maneuvers are either 3 or 5 sec. in duration and correspond to various turn and climb/descent commands.  The MAV performs the first of these maneuvers, and then a new optimal control sequence is computed.  Each time this optimization is performed, it uses the current estimate of the three-dimensional environment to enforce obstacle avoidance constraints.    Figures 3 and 5 depict two different views of the total flight path along with the estimated geometry of the scene.  For comparison, Figures 4 and 6 depict the total flight path in the virtual environment.  The adaptive estimation process and the receding horizon control are demonstrated in the movies “flight_learning” and “flight_learning_overhead”.  These movies provide two different views of the learning and decision processes at various points in the flight.  In the movies, the red line denotes the prior flight path, the yellow line represents the optimal path chosen by the receding horizon controller, and the green line depicts the maneuver that is actually performed by the MAV.  Note that the entire environment is not mapped during the flight, only regions that were viewed by the onboard camera.  The movie “flight” illustrates the MAV flight as viewed by the onboard camera, and the movie “flight_fpts” shows the onboard camera images along with the tracked feature points.  The movie “flight_overhead” provides an overhead view of the MAV traveling through the virtual environment.  As a final example, Figure 7 shows the results of a similar flight that makes use of a pre-computed map of the environment.  This map was learned using images from a downward-pointing camera obtained during a simulated flight over a region of the environment.  The overhead camera viewed more of the environment, resulting in a more complete representation of the stadium and some of the surrounding buildings.  This example illustrates the ability to incorporate pre-existing information about the environment into the vision-based control framework.      

Figure 3:  Overhead view of the flight path with estimated geometry.

Figure 4:  Overhead view of the flight path in the UF campus virtual environment.

Figure 5:  Three-dimensional view of the flight path with estimated geometry.

Figure 6:  Three-dimensional view of the flight path in the UF campus environment.

Figure 7:  Flight path using estimated geometry from a previous overhead flight.