Article by: Paul Clarke, QinetiQ, UK (pdclarke@qinetiq.com)
Introduction
Generating terrain databases for Synthetic Environments (SE) has historically been a complex and expensive process. Over time, users have demanded increasingly realistic, feature rich and accurate SEs covering larger geographical areas. as the complexity of terrain databases increases, so too does the probability of errors within them [1].
Two types of anomalies can affect terrain database quality:
·Integrity errors
Integrity errors occur within a database when certain features are not represented in a feasible way (i.e. as they would appear in reality). Typical examples include buildings placed on roads, water flowing uphill, holes in the terrain and roads with implausibly steep slopes.
·Correlation errors
Correlation errors occur between different database representations. An SE usually contains a number of subsystems such as the visual out-of-the-window database and the Computer Generated Forces database. If features in one database are represented differently in another (or are missing altogether) then this can have serious implications on the usability of the SE. For example, a road present in a 2D map display but missing from the 3D visuals could impair a user’s ability to navigate effectively.
Until recently, the quality of databases was inspected manually; this is clearly unfeasible for anything but the most simple SE. A highly desirable approach is therefore the automated assessment of integrity and correlation which can provide both exhaustive and quantitative evidence. Two such efforts are the Synthetic Environment Evaluation – Inspection Tool (SEE-IT) from the Institute of Defense Analyses [2] and a new Verification and Validation (V&V) tool – Venator (which is Latin for hunter).
Venator
Venator is an extension for ESRI’s popular Geographical Information System (GIS) ArcGIS™ [3]. Leveraging existing GIS software offered a cost-effective solution and afforded us scalability, a comprehensive visualisation environment, support for various co-ordinate systems and the ability to import numerous data formats.
Analysing terrain databases using Venator involves a process with four separate stages:
·Data input
Given Venator’s remit for importing everything from source data to runtime databases, a standard file format for storing geospatial data was required. ESRI shapefiles® were chosen as they are an industry standard and open format. Venator currently supports DTED, VMAP, OpenFlight™, SEDRIS, Close Combat Tactical Trainer (CCTT), Combined Arms Tactical Trainer (CATT) and the Compact Terrain Data Base (CTDB).
·Integrity and correlation processing
A number of integrity and correlation tests which check for specific anomalies and errors have been implemented. Each test is a distinct plug-in with its own graphical wizard that guides the user through the various parameters required for analysis. Appropriate test parameters often vary from database to database and feature type to feature type; an element of ‘trial and error’ is inevitable. This modular plug-in architecture makes it very simple to add new tests as and when needed. For example new, database-specific tests, are sometimes required. Such tests can be constructed rapidly using existing tests and code as templates. For convenience, a batch mode is available so several tests can be set up and run sequentially. This facilitates an efficient use of time as tests can be executed overnight and analysed the following day.
The raw data produced by the integrity and correlation tests can be output in a number of user-defined ways: an error layer (which contains all the input features with potential anomalies); an aggregated error layer (where nearby anomalies are aggregated into single features); a summary statistics table, a tabular version of the results (CSV file) and, in the case of correlation tests, a graph. Aggregating anomalies is particularly useful when dealing with an error layer containing thousands of features, so clusters of errors can be identified.
·Results analysis
Producing raw test data is in many ways the easy part – the real challenge is in interpreting the results to extract meaningful conclusions about the ‘quality’ of the database. Although the output from some tests is straightforward (e.g. a feature either conflicts with another feature or it doesn’t) most require an element of human reasoning and intelligence. As it not feasible to manually inspect every potential error a utility, the feature scanner, was created to filter the anomalies based on one or more feature attributes. This ensures the most serious anomalies are dealt with first. Each feature can be cycled in turn and the view focuses on that particular feature so the potential error can be assessed in the context of surrounding features.
·Presentation of results
The final stage of the Venator process is presenting the results of the analysis in a format appropriate to the interested parties. Providing a spreadsheet which documents the location of each and every integrity and correlation error (which could run into the hundreds of thousands) is unlikely to be of interest to a high level manager charged with procuring a new terrain database. However, a filtered version may be of use to the database engineers in isolating certain problems.
Standard written reports are just one of the ways for distributing results. It is also possible to output labelled error maps; interactive online maps (viewed in Internet Explorer) using the EasySVG™ [4] extension and spreadsheets of raw or filtered error data.
UKCATT Case Study
The UK Combined Arms Tactical Trainer (UKCATT) is an advanced virtual training system consisting of two hundred manned simulators split between sites in the UK and Germany. CATT supports large-scale exercises played out on detailed terrain databases typically 100x100 km in size. Each database comprises several representations: visual (out-of-the-window); Semi Automated Forces; Plan View Display and paper maps which must all be correlated. QinetiQ was tasked by the Defence Procurement Agency (DPA) to support the test & acceptance of CATT and verify that the terrain databases fulfilled the Statement of Requirements. Under this project, Artemis, the Venator capability was utilised to provide automated analysis. In addition to this formal role, several interim analyses were undertaken and a good working relationship established with the contractor Lockheed Martin Information Systems (LMIS). When new versions of the terrain databases were generated, an interim analysis was used to document and verify the improvements made by LMIS. This new approach to test & acceptance, adopted by the DPA, increased confidence in the CATT terrain databases by replacing qualitative judgements with quantifiable evidence and they were successfully accepted in July 2002.
Conclusion
Terrain databases will never be perfect, given the complexity of the processes used to generate them. It is therefore important to gain an insight into the types of errors present and how widespread they are. Failure to do so could result in significant problems when the SE is finally deployed which may severely limit its usefulness. Understanding integrity and correlation is equally important when reusing legacy terrain databases or interoperating several SEs. The manual inspection approach is no longer viable and new semi-automated inspection is the only reasonable alternative.
Tools such as SEE-IT and Venator are ultimately designed to help improve quality and reduce the cost and time involved in rectifying the types of errors they identify. Utilising GIS technology and industry standard formats means that a wide range of data can be analysed from source data to runtime databases. Integrity and correlation testing is a technically demanding and non-trivial task which currently augments human intelligence rather than replaces it.
If the issue of quality is ignored, terrain databases will continue to be affected by broken bridges, flying tanks, trees blocking roads and all the other problems currently encountered in SEs today. We suggest automated V&V approaches are employed as a matter of course in order to replicate the success of systems like UKCATT.
References
[1] Richbourg, R. and Stone, S., “Towards the Production of Syntactically and Semantically Correct Synthetic Environment Databases”, In Proceedings of the Spring Simulation Interoperability Workshop, March 2003, 03S-018.
[2] Synthetic Environment Evaluation – Inspection Tool, http://tools.sedris.org/seeit.htm, accessed July 2004.
[3] ESRI website, http://www.esri.com/arcgis, accessed July 2004.
[4] EasySVG™ website, http://www.easysvg.com, accessed July 2004.
Acknowledgements
This work has been sponsored by the MoD Directorate of Analysis, Experimentation and Simulation and funding from Package 3A of the MoD Research Programme.
To view the entire paper visit: www.sisostds.org/doclib/doclib.cfm?SISO_RID_1005426
|