The Precision Loss of Double-to-Float Conversion

Incorrect display

Posted by UG on January 22, 2023

Problem

The graph below illustrates the difference in the display scene beteewn our application and AutoCAD. We can see the point-value issue here. image

Debugging

This curve are composed of B-splines, so I remove most of them and only keep small portion of the curve to simplify debugging. The accompanying figure compares the simplified curve in AutoCAD and our application, it clearly illustrates the problematic display. image

I have no idea unitl examining the X/Y value of one of the points, the value is large. image

And another problem is the gap between any two points is extremely small. image

As before rendering, we have the double-to-float function, combining the large-value and small-gap problem we talk about above. It may raise the precision issue, resulting in rendering two line in scene, not a curve.

void dataTransfer(const Point p0, const Point p1){
    data.push_back((float)p0.x);
    data.push_back((float)p0.y);
    data.push_back((float)p1.x);
    data.push_back((float)p1.y);
}

In this case, from the first to the eighth point, the value remains the same at 884828.875 by the conversion loss. However, from the ninth to last point, the value remains at 884828.9375. image image

Solution

A simple solution for this issue has yet to be found. Using double precision in rendering is not currently feasible. To resolve this, the X/Y values can be kept within 10000 by translating. The graph below illustrates the translated curve, which now accurately display the data. image