Knowledge Base

Welcome to The Carlisle Group's Knowledge Base.

Search or Ask a Question

This is a keyword search that looks for matching articles that contain one or more words specifically by a user in articles’ tags, title and content. Admin writes brief intro content here via admin end. If you are unable to find an answer to your issue, please submit an issue here.

Back to Articles List

CAS floating point precision and IEEE 754 | CAS

Added: 05/03/2012; Last Modified: 12/29/2021; Visits:2,176

CAS Adheres to the IEEE (Institute of Electrical and Electronics Engineers) 754 technical standards for machine based floating point arithmetic. This standard is used universally by most machine based tools, including Excel. This standard makes it much faster for machines to solve large problems with floating point numbers, and because of that, there are a few cases where numbers cannot be tied out to a very long decimal place. 

If you are using CAS to tie out to a number with several decimals, and showing a difference, it might be advisable to convert the number to an integer, run the calculation, then convert it back. 

An example of this, where you have 4 decimal places for a computed balance, could be to use the following workaround:

(total (1000 times CURBAL)) div 1000

 

Multiplying computes the total on the integer, not the floating point value, and then dividing converts it back to the proper place. 

 

Keywords: rounding formatting tying out, accuracy decimal floating point