Connexion rapide:  

Forum: VirtualDJ Technical Support

Sujet about integer precision scripts

Ce topic est ancien et peut contenir des informations obselètes ou incorrectes.

NicotuxHome userMember since 2014
instead of result of math operation we can get
inf or -inf or nan or -nan(ind) or maybe other
is there a way to detect when an over/undeflow appends ?

about integer precision ...
strange leak of precision over 8 digits
11111111 * 10 + 1 = 111111112
111111112 */ 0.1 = 0
this seems to indicate that script are using float (32 bit 4 bytes ±1.18 x 10^-38 up to ±3.4 x 10^38 6-9 significant digits)

this is more than enough for GL textures and operations over sliders 100% or range 0..1 or MIDI byte operations
but 16bit fails sometimes, this is not enought for many other things i.e.: time/date, filesize, bitrate starting 192K

specially 64bit version may not have this limitation anymore
 

Posté Sat 22 Jun 19 @ 6:59 am
AdionPRO InfinityCTOMember since 2006
If the arguments are integers then the result will be integer (32-bit signed)
If either of the arguments is floating point, then the result will indeed also be calculated in 32-bit floating point.
 

Posté Sat 22 Jun 19 @ 9:01 am


(Les anciens sujets et forums sont automatiquement fermés)