Presented at the XII International Conference on the Use of Computers in Radiation Therapy (XII ICCR), May 27-30, 1997, Salt Lake City, UT.

Refer to the presentation for illustrations of some of these principles.

3D visualization techniques are commonly employed in radiation treatment planning (RTP). The resulting pictures are often pretty, and a tendency exists to view these as inherently correct, since they look so much like the reality we think they accurately represent. However, it can be very difficult to know what is really being displayed and whether it is being perceived and interpreted accurately by the viewer, as the visualization community is well-aware. The visualization process can suppress information about the original data quality, add information not present in the original data, and lose information. Different visualizations of the same data can appear to disagree. Tradeoffs are often made between speed and accuracy, possibly without understanding the errors. All of this can result in incorrect assumptions about the structure or content of the original data or the meaning of the visualization, but the observer is often unaware of inaccuracies. It is important to understand the significance of these issues in the overall RTP process. Whether any of them really matter depends on the goal of the visualization and the difference between the actual data and what is being conveyed to the observer. If this error could be measured, more meaningful visualizations could be created, leading to more accurate and consistent interpretations of the data. Tradeoffs could be assessed during RTP system development, different systems could be compared, and optimal visualization techniques could be investigated. The goal of this paper is to initiate discussion on this problem and on possible solutions. First, some common RTP visualization issues are analyzed in an initial effort to characterize the problem. Then, one possible method of modeling and evaluating these issues is presented as a strawman proposal. Only the surface of these issues is scratched, but hopefully a framework is provided for continued investigation.

General Visualization Issues

"Visualization is the process of mapping numerical values onto perceptual dimensions" [1]. Ideally, our interpretation of the result should effectively and accurately represent the original data. Two primary issues can obscure this goal:

1. the process used in creating the visualization, and
2. the process by which the display is perceived and interpreted.

Visualization can suppress information about original data quality, for example, by interpolating or otherwise representing missing data [2]. Visualization can add information not present in the original data. Mapping dose data to a rainbow color map gives the erroneous impression of values clustered in discrete bands [3]. Sometimes, more of the information bandwidth in the visualization comes from the modeling than from the actual data [4], such as smooth shading of a polygonal isodose surface, which gives a false impression of the granularity of the original dose matrix. Even a very coarse dose matrix can generate rather smooth isosurfaces. Visualization can hide information. Mapping dose data to a rainbow color map can hide significant detail within each color band. One might argue that polygonal isodose surfaces should be displayed with flat shading to accurately portray the dose matrix granularity. However, if an algorithm other than marching cubes is used or the surface has been simplified, the granularity might now appear more coarse than it really is. Different visualizations of the same data can disagree. A dose volume histogram (DVH) based on dose voxels might disagree with the display of the structure surface with dose mapped to it or with a dose isosurface on top, since the latter dose mappings are likely based on dose points.

In any computation, a tradeoff can exist between performance and accuracy. To balance this tradeoff, one often simply assesses the error between the correct answer and the computational result. However, with visualization, both the correct answer and the computational result can be difficult to determine. The computational result is our interpretation of the visualization, while the correct result is what our interpretation of the raw data would be without the visualization. This is clearly not a simple calculation. Tradeoffs are nevertheless being made in RTP systems, possibly without an understanding of the errors and almost certainly without communication of these issues to the users.

Further, we are prone to have a false sense of security due to the nature of RTP visualizations. These generally attempt to parallel the physical meaning of the data. Examples include dose volume information displayed as a 3D isodose surface, creation of a structure surface from a collection of CT contours, or an arbitrary slice through a CT volume. Such visualizations, which appear to be direct mappings from the original data space, can lead one to view them as inherently "correct" [4]. This is in contrast to less intuitive techniques, such as using parallel coordinates to display multivariate relations.

These issues are not specific to RTP. The young field of scientific visualization still wrestles with them [5 , 6 , 7 , 1]. Little has been done to define a good visualization or visualization system; to provide guidance in making reliable, accurate visualizations; or to provide benchmarks for designing, validating, or comparing visualization products [7]. However, our task should be much more tractable since it is very well-bounded, with well-understood goals.

It is important to understand the significance of these issues in the overall RTP process. Whether any of them really matter depends on the goal of the visualization and, in the end, the difference between what is desired to be conveyed and what is actually conveyed to the observer. Essentially, we are looking for an error measurement, albeit a complex one! The first step must be to understand the problem clearly. Then we can look for ways to measure and communicate this error.

RTP Visualization Issues

Some common RTP visualization issues are now discussed to begin to characterize the problem. In each case, consider how these issues can be characterized, evaluated, quantified, and communicated, such that one can understand the possible error in the results and compare results from different approaches.

Isosurfaces and Surface Simplification

In the commonly used marching cubes algorithm [8], calculation time depends on the size of the data set, while the complexity of the surface depends on the voxel size. Thus, even a very smooth (low spatial frequency) surface can consist of a large number of triangles, resulting in reduced interactive performance. Both problems can be simply addressed by first spatially filtering the volume data, mapping to a lower resolution 3D grid. Calculation time will decrease and surface complexity will decrease, yielding increased interactivity, but at the cost of information loss. For dose information, this can be dangerous. Note, the resulting isosurface will likely appear even smoother than the original.

Newer, more efficient approaches to isosurface generation can reduce calculation time and resulting surface complexity (e.g., [9]). However, these are research topics and might not be available in the chosen visualization package, or limited development resources might preclude these as options.

One safe approach is to accept the calculation time, then apply a surface simplification algorithm to reduce the number of triangles in the mesh (improving interactivity) while maintaining a "good" approximation to the original geometry. How good? There have been many approaches to this problem, with different error measurements and tolerances. In a recent powerful approach from IBM [10], the original volume is fully preserved and a maximal surface error is rigidly enforced at each vertex, while favoring equilateral triangles. However, this can take longer to run than less rigorous algorithms.

Thus, the following tradeoffs exist.

1. Use the original data, and display as it is. Surface generation time and interactivity may be too slow, but information is not lost.
2. Spatially filter the volume data up front. This speeds up both surface generation and interactivity, but information is lost. The information lost is proportional to the speed-up.
3. Use the original data, followed by surface simplification. Surface generation is slower than (1), but interactivity speeds up. Accuracy depends on limits of the surface simplification algorithm.
4. Use the original data, followed by "accurate" surface simplification. Surface generation time is even slower than (3), but with similar interactivity speeds; accuracy, however, will be much better defined.

When observing a given isosurface, how can one know which tradeoffs were made, and what is the possible error? When visualizing dose, these issues can be critical.

Surface Shading

Surfaces are usually displayed with smooth shading. It not only looks prettier than flat shading, but the latter can cause reduced interactivity. Smooth shading both adds and hides information. For example, with marching cubes, the surface triangle size directly indicates the data voxel size. Displaying these triangles with flat faces clearly shows the original volume data granularity. Smooth shading adds information by interpolating values at the display pixel resolution, thus hiding the volume data granularity. Of what value is this added information?

Consider two smoothly shaded isosurfaces, one calculated from the original volume data, the other from grossly reduced volume data. Both would appear smooth. Flat shading would clearly show the difference. However, if the surface has been simplified, the original voxel information is lost anyway, and flat shading would incorrectly portray the original dose matrix as more coarse than it was. When looking at a surface, it can be difficult to assess the data used in its calculation.

Surface Generation from Contours

Two common approaches are employed to generate a surface from a set of contours. One is to connect contours between adjacent planes (e.g., [11]). Accuracy issues can arise from the connection algorithm (e.g., how it handles structures that branch or that come and go) and from contour or surface simplification methods. The second approach is to render (fill) the contours, stack these images to form a volume, then create an isosurface. Accuracy issues can arise from the rendering resolution, the isosurface algorithm, and surface simplification methods.

The previously discussed isosurface and surface simplification issues and tradeoffs apply to both of these approaches. In addition, the following questions arise: How well does the resulting surface actually "fit" the original contours? What is the best way to measure and express the possible or actual error? Is the error data-dependent? Such error can be important when viewing the tumor surface.

Volume Data

In general, RTP volume data can exist in one of three forms:

1. A completely regular grid with regular positions and connections (e.g., a set of CTs at constant table position increment, each with identical XY resolution and field of view [FOV]);
2. A stack of completely regular and identical XY grids, but at variable Z spacing (e.g., the previous CT set, but at nonuniform table positions); and
3. A stack of completely regular but different XY grids, at variable or constant Z spacing (e.g., the previous CT set, but with a different FOV or resolution for some images).

For certain volume data operations, regular grids can have much lower memory requirements and can be substantially faster to work with then irregular grids. Such operations include mapping the volume data to arbitrary planes or surfaces and isosurface generation. This can give incentive to regularize the volume data on input, for example, by remapping all CT or dose planes to the same XY resolution and FOV, or by remapping variably spaced CT or dose planes to a constant Z spacing. The following analysis for CTs applies to any volume data.

In remapping XY, there are two choices. To reduce information loss, all images might be remapped to the pixel size of the highest resolution image. This can cause a data explosion. Consider a stack of 512x512 CTs. If just one CT has an FOV of 1/2 the others, remapping would produce a stack of 1024x1024 CTs. However, if the smaller FOV CTs are not of interest, they could simply be removed from the data set, with no remapping required. To prevent such data growth, one can instead remap to the pixel size and FOV of the lowest resolution image. This would cause loss of information in the CTs with 1/2 the FOV. This would be a nonissue if those CTs are not of interest but could be critical if the higher resolution CTs were specifically obtained for more detail in the target area. Note, even when remapping to the smallest pixel size, interpolation will occur and information can still be lost.

In remapping Z, the issues are similar. To reduce information loss, one might interpolate CTs at regular Z spacing equal to the minimal Z spacing of the original set. This can also cause a data explosion. Consider one CT at 1/3 the spacing of all the others. Remapping would triple the data size. In any case, even if most of the original CTs are regularly spaced, it is quite possible that most of the interpolated set will not align with them, resulting in loss of information. On the other hand, if the irregularly spaced CTs are not of interest, they could simply be removed, with no need to regularize.

The impact of the chosen approach and its significance depend entirely on (1) the structure of the original data, (2) the target of the visualization, and (3) the goals of the visualization. Perhaps both the original and regularized data should be carried along, each for different purposes. The observer is usually unaware of these issues and thus has no idea of the accuracy or meaning of what is being observed.

Voxel versus Point Values

Whether volume data is interpreted as voxel data or point data has an impact on how well various visualizations will agree. For example, dose isosurfaces are usually interpolated through a volume of dose points. However, in DVH calculations, dose voxels might be employed, where the dose pixel is considered constant through the slice thickness. Now consider an ideal structure, precisely filled with 100% dose voxels, with 0% dose voxels outside. The DVH will indicate that 100% of the structure receives 100% dose. However, a 100% dose isosurface will lie entirely inside the displayed structure surface, since it will pass through the 100% dose points, and not at the boundaries of the 100% dose voxels. A 50% dose isosurface might be required to fully "cover" the structure. The same problem can occur in mapping dose values onto the structure surface, since each surface point will get an interpolated value between the inside and outside dose point, not the value of the dose voxel it passes through. Most of the surface might appear to receive 50% dose.

To bring these into agreement, one can employ voxel data in the isosurface and dose mapping steps. The dose isosurface would be the boundary of the dose voxels (like stacked sugar cubes). This would require something other than marching cubes and would create a much more complex surface (causing reduced interactivity). It would also be difficult to correctly display voxel dose data mapped onto the structure surface, given the triangulated surface facets.

Color Mapping

The mapping of data onto a color scale should be tuned to the goal of the visualization, the type of data and its spatial frequency, and human visual perception if we expect the user to accurately interpret the information [3]. For example, the commonly used "rainbow" color map both creates erroneous information (values appear to be clustered in discrete bands) and hides real information (hiding detail within those bands). Allowing the user to create custom color maps is equally if not more dangerous. When observing apparent structure in color mapped data (such as dose values), does this represent the data, or is it just a color mapping artifact? One solution is described in [3], which provides optimal color maps based on data type and spatial frequency, the user's task, and properties of the human perceptual system. We need to characterize the color maps most appropriate for common RTP visualization tasks.

Missing Data

How missing data is handled is important. Interpolation can lead to misinterpretation of data, but leaving it blank, black, or transparent can distract the viewer from more relevant information. In [2], a color map is used for valid data, a grayscale for missing data, with the boundaries matched via luminance value. This does not distract the viewer at the edges, yet clearly shows valid and invalid data.

A Proposed Model

A method of modeling and evaluating the RTP visualization issues is now presented. Such a tool could be utilized to assess tradeoffs during RTP system development, to compare different RTP systems, or to research optimal visualization techniques. This is a strawman proposal, offered primarily to help understand and structure the issues. In the end it may serve only to show how not to do this. There are many related efforts at modeling and evaluation in general visualization, computer graphics, image processing, and computer vision, such as [12 , 13 , 5 , 14 , 7]. This proposal is a first step in applying some of these ideas to the RTP problem, but much more investigation is required.

Modeling the Visualization Process

An RTP system can be viewed as two cascaded processes:

1. the creation of a treatment plan, followed by
2. the visualization of that plan.

The first process is outside the scope of this paper, but its output data, such as the dose volume, reference points, beam parameters, and block design, form the input to the visualization process.

The visualization process is modeled in an abstract, object-oriented manner (e.g., [12]). Each large-grain visualization operation becomes an object with well-defined inputs, outputs, and data transformations (transforms). For example, an isosurface object might take volume data and isovalue as inputs, produce a triangulated surface as output, with the marching cubes algorithm as the transform. Since the input and output data are objects as well, "operation" objects will generally be called modules, such as an IsoSurface module or SurfaceSimplify module. Note, the final visualization module will usually be the HumanObserver.

Each visualization module must also provide some measure of error, confidence, or quality of the output data. This is not necessarily a simple value. For example, a SurfaceSimplify module would provide indication of how the new surface relates to the original. As in [10] this would be

1. a guarantee that volume is preserved,
2. a guarantee of maximum vertex error, and
3. an absolute error value for each vertex.

This quality measure will be termed the Quality Metric (QM). The effect of QMs must be propagated through all modules in the visualization process. Each module must calculate its output QM by considering the QM of its inputs and how these are impacted by its own transforms. It will then be possible to measure, evaluate, and compare the results of visualizations.

For general scientific visualization efforts, this type of modeling is a formidable task and not well-understood, especially considering how such "quality" issues compound, cancel, or otherwise interact (e.g., [5 , 14]). However, the RTP problem is very well-bounded with a rather small set of clearly defined goals, hopefully making it more amenable to such a solution. The process of developing this model will be iterative and evolutionary. As modules are defined, the need to modify or create new data types or QMs may arise. As data types or QMs change, previously defined modules may have to be modified. The overriding constraint must be to keep things as simple as possible, but no simpler (paraphrasing Einstein).

The necessary modeling tasks are now briefly introduced, focusing primarily on the more difficult QM modeling.

Transform Modeling

Transforms (data transformations) are modeled at a high level of abstraction. For example, a Slice module which extracts a 2D slice from 3D data by holding one element of the position vector constant would be trivial to model. Similarly, a spatial filtering module can be fully characterized by its impulse response or a simplified approximation. However, consider an IsoSurface module based on marching cubes. How can the resulting surface be modeled in a simple way? What really needs to be modeled? Following are some examples of the issues.

How large is the surface? This information is necessary to determine what can be resolved on this surface, possibly based on the number of display pixels that render it, or the angle it subtends in the human visual FOV. For example, an isosurface accurate to 1% may be of little value if the end observer can only detect a delta of 10% in the surface features.

What size are the triangles? Triangle size is necessary to understanding the resolution of data mapped onto this surface. For example, mapping a high resolution dose volume matrix onto triangles that are 10 times the size of the dose voxels can result in substantial information loss, even though smooth shading among vertices would hide this.

Other possibly important aspects include number of triangles, surface normals, whether data is position (vertex)-dependent or connection (triangle)-dependent, and data type. These issues must be thought out for each relevant visualization operation.

Data Type Modeling

This is the simplest modeling issue and would proceed in parallel with transform and QM modeling. Fundamental data types can be seen in any of the common visualization packages, and from this, simplified versions for our purposes can be created.

Quality Metric (QM) Modeling

Whereas the transform describes the changes to the data, the QM provides ancillary information about the quality of that data (e.g., an error bound on a data set or on each data element or other descriptive information). QM information will exist purely as attributes on the transform results. One should be able to remove all QM information and still have a workable model (though far less useful), whereas removing the transform information would break the model.

Since each transform has an impact on the QM of its outputs, the value of the QM is clearly transform-dependent. However, the structure of the QM must be purely data-dependent. This allows each module to process the QM independently, regardless of where it comes from. Following are some issues that the QM must handle, as well as some examples.

Non-information that is added. This could be due to smooth shading, color mapping, interpolation across missing data, or interpolation at higher resolution than original data.

Information that is lost. This could result from smooth shading and its relation to viewing transformations and lighting, color mapping, data filtering, interpolation, or masking by non-information.

Error bounds on valid information. These are often based on a mathematical transform (e.g., calculation error due to the algorithm or data type limitations). However, for the HumanObserver module, they will be based on visual perception issues.

The intrinsic value of a visualization method. Different methods will have different impact and value. For example, displaying a dose volume matrix as raw numerical data is completely accurate but worthless as a visualization tool. Displaying it as an isosurface is more informative but perhaps less accurate. Visualization methods could be evaluated and ranked in terms of suitability [14 , 5], based on data characteristics, goals of the visualization, required accuracy, and visual perception issues.

Following are some examples. Consider the process of regularizing a dose volume data set (discussed in the first part of this paper). The QM should indicate how the regularized data differ from the original. This cannot be expressed as a simple value. Consider that 44 out of 45 planes might align exactly with the originals, with just one that is interpolated between two originals. If it lies on the edge of the study, it might not matter. If it lies in the tumor volume, it might matter a great deal. This information can be captured in the QM by assigning an error value per plane; 44 planes have zero error, one plane has non-zero error. Later analysis of the particular visualization tasks and targets will then determine the significance of this error.

As another example, consider a dose isosurface generated from marching cubes. Each facet represents the physical location of that dose value and corresponds to a dose voxel. If this surface is simplified, the resulting facets still represent the location of that dose value, but now there are possible positional errors. The QM can represent this with a position error, but what does that say about the possible dose error? One really wants to know, for a given facet representing a particular dose value, how much that dose can be in error. This might require mapping the position error back to the original dose matrix and using its gradient to determine an error bound on the dose value represented by the isosurface. Further, these new facets no longer correspond to original dose voxels. The QM could include an attribute that specifies relationship of facets to input data. Prior to simplification, the facet relationship would indicate input voxels. After simplification, it would be null. If a subsequent visualization employed flat shading to show original voxel size, this attribute would signal that the facets no longer represented anything meaningful.

Understanding how to propagate QMs, how they compound, cancel, or otherwise interact can be tricky. Simply changing the order of operations can have a big impact. For example, interpolating between vertex colors in rendering hardware can give different results than interpolating data values and then mapping to colors [5]. These issues have to be thought out.

Example Use of the Model

In the end, if the accuracy requirements of the final data interpretation can be specified, the modeling tool will determine if a proposed visualization will suffice. A simple scenario will demonstrate this. Assume the goals are to

1. visualize dose within a specified accuracy and
2. to resolve dose changes of a particular delta.

One visualization of this might consist of

1. spatially filtering the original dose data,
2. generating an isosurface,
3. simplifying,
4. smooth shading,
5. displaying, and finally
6. observing.

The analysis would determine the error bound of the dose value represented by the resulting isosurface by considering steps (1) through (4). Even if this error is within the desired tolerance, the process is not complete. The analysis must also consider the impact of display and observation of that isosurface, such as whether the viewing angle and pixel resolution are sufficient for the observer to detect the desired changes.

Another visualization might consist of

1. generating a structure surface from contours,
2. simplifying,
3. mapping dose data onto it,
4. applying a color map,
5. smooth shading,
6. displaying, and
7. observing.

Here, an additional constraint might specify that the structure surface align with the original contours within some tolerance. The surface error bound would be determined from steps (1) and (2), with the dose error bound determined from steps (3) through (5). Then, as above, consideration must be given to the impact of display and observation, such as whether the color map will allow the observer to detect the desired dose changes.

Summary

The issue of accuracy in RTP 3D visualization techniques was presented, pointing to the need for methods of measurement and evaluation. A number of common RTP visualization issues were analyzed in an initial attempt to characterize the problem. A strawman proposal was then presented, describing one approach to modeling and evaluating these issues. Only the surface of these issues has been scratched. The goal was to initiate discussion on this problem and possible solutions.

References

[1] Marchak et al. The psychology of visualization. In IEEE Visualization 93, pages 351-54, 1993.

[2] Twiddy et al. Restorer: A visualization technique for handling missing data. In IEEE Visualization 94, pages 212-16, 1994.

[3] Bergman et al. A rule-based tool for assisting colormap selection. In IEEE Visualization 95, pages 118-25, 1995.

[4] Roux et al. Visualization in medicine: Virtual reality or actual reality? In IEEE Visualization 94, pages 396-99, 1994.

[5] Globus et al. Evaluation of visualization software. Computer Graphics, 29(2):41-4, May 1995.

[6] Gershon. Breaking the myth: One picture is not (always) worth a thousand words. In IEEE Visualization 96, pages 441-43, 1996.

[7] Rushmeier et al. Metrics and benchmarks for visualization. In IEEE Visualization 95, pages 422-26, 1995.

[8] Lorensen et al. Marching cubes: A high resolution 3d surface construction algorithm. Computer Graphics, 21(3):163-69, July 1987.

[9] Shekhar et al. Octree-based decimation of marching cubes surfaces. In IEEE Visualization 96, pages 335-42, 1996.

[10] Gueziec. Surface simplification inside a tolerance volume. Technical Report RC 20440, IBM, March 1996.

[11] Boissonnat et al. Three dimensional reconstruction of complex shapes based on the delaunay triangulation. In Biomedical Image Processing and Biomedical Visualization, pages 964-75, 1993.

[12] Butler et al. Visualization reference models. In IEEE Visualization 93, pages 337-42, 1993.

[13] Hildebrand et al. Towards a visual computing and communication reference model. Computers & Graphics, 19(1):144-49, 95.

[14] Arndt et al. Design of a visualization support tool for the representation of multi-dimensional data sets. In Visualization in Scientific Computing, pages 190-204, 1995.