A neat trick many people aren't aware of is that you can treat binary floats as saturating fixed point, subject to some qualifications (generally the next larger float type can represent any given fixed). Float operations internally are "just" fixed point ops with some normalization steps and rounding bits on each side, so if we use a float type with enough mantissa bits to hold the fixed point value all we have to do is mask off the extra precision to get back to fixed point. This similarity to fixed point is exploited in some modern NPU hardware by storing only one exponent for an entire block of floats, with a wide fixed point unit doing the actual work, a.k.a block floating point.
This hack has some interesting advantages. Float to integer is still only a few cycles, the masking is one line of libm functions, you get better (and dynamically selectable!) precision, it has gradual underflow and overflow, you can write numeric code like usual, and normalization is automatic.
Usually, if you know enough about your algorithms to select an appropriate float alternative, you also know enough to fix your float code and that's what you should actually do.
That said, some of these aren't alternatives. Symbolic computation is a different thing entirely. Interval arithmetic can be built atop floats (e.g. IEEE-1788) and has its own zoo of unintuitive behaviors. BCD is better called a historical artifact than an alternative these days.
It's really just rationals and decimal floats in this list, which probably don't solve the issues you have if you're considering float alternatives.
I think people with certain backgrounds look at stuff like x86 BCD opcodes being removed in the move from x86-32 to amd64 and think "must be because noone uses it". Other backgrounds know that ain't so.
You can't "fix" floating point code if you are looking for deterministic answers. You just have to use other data types to handle money or complex mathematical operations like 0.2+0.1, no ifs and buts.
Floats are deterministic, but I get what you mean. Let's discuss what's meant by the result of a complex calculation. 0.1+0.2, or sqrt(2), or whatever.
1. Do you want your result to exactly encode the answer without rounding error? No fixed precision type can provide this in general, so you're stuck with symbolic approaches. If you can bound things (usually difficult), maybe you can get away with non-symbolic approaches.
2. Do you want a sensible numeric answer? This is what floats (and many other systems) give you. The definition of "sensible" is inherently tricky here and there's not a definition universally appropriate to every possible computation.
So let's return to 0.1+0.2=0.3000...1 specifically. There's two common ways to think of an encoded float. One is as the directly encoded value, as you're doing. Another way is to think of it as an interval of real numbers between the next lowest and highest intervals. Under this latter interpretation, it makes sense to discuss shortest decimal string within the interval, 0.3 in this case. There's no ambiguity because each real lives in exactly one interval. This is what algorithms like dragon box do for float to decimal string conversion.
What decimal floats give you is an encoding that tracks significant digits, where every decimal string exactly corresponds to a midpoint of an interval of reals. They do this at the cost of space, speed, and complexity. You don't get an escape from the fundamental issues of fixed precision types like rounding error, numerical sensitivity, precision loss, etc. I don't think that tradeoff makes sense for most algorithms in most contexts.
The benefit of sticking with floats is that lots of smart people have spent countless hours trying to give non-experts a "good enough" path through the untamed wilds of numerical analysis, tooling to help them when they get lost, tribal knowledge to point out the edge cases, and it's almost universally supported in hardware. By all means you should go wandering off the trail, but fully understand what you're doing and why beforehand.
> Another way is to think of it as an interval of real numbers between the next lowest and highest intervals. Under this latter interpretation, it makes sense to discuss shortest decimal string within the interval, 0.3 in this case.
Note that the result of 0.1+0.2 does not lie in the interval containing 0.3, which is was confuses most people. The issue is that there is some imprecision in representing 0.1 and 0.2 too, and that compounds when summing, resulting in something that does not actually correspond to 0.3 (hence the classic 0.1+0.2!=0.3)
But, as I understand it, 1+2=3 in all of these senses using floating point; as long as you don’t go outside of a certain very large range, they are really a superset of integers.
That makes me think that I can just plan ahead by storing the number of cents instead of dollars, or a “hack”, and then it makes me wonder why the format even requires me to do that.
0.2+0.1 with floating point numbers _is_ deterministic, as you'll always get the same answer.
I suspect you might instead mean exact calculations/answers (in the example above, neither 0.1, 0.2 nor 0.3 have exact representations using floating point numbers).
And just to be clear, there are non-determinism-like issues with floating point numbers, but those are much rarer/niche and _can_ be fixed. For example parallel summation depends on the order the summation was made, so non-determinism in the parallel implementation ripples through the summation result. Some non-basic operations (e.g. trigonometric operations) have platform dependent implementations with different roundings, so you might experience different result based on the platform you're on.
CPUs have many different configuration bits to configure floating point rounding, flushing denormals, etc. which mean that in practice anything that relies on floats being deterministic has the stability of a house of cards.
> Floating point is deterministic, what are you talking about?
Order of operations can change a result, for example. I suspect you mean that the algorithm never changes. While op means that mathematical operations which most folks would expect to be reliable are not.
There are enough problems for a 44 page paper titled "What Every Computer Scientist Should Know About Floating-Point Arithmetic"[1] I don't quibble on the language because I know what people mean.
Most folks won't encounter most of the issues, generally. But expose your code to a large enough dataset, or be like me and write a CAD/CAM system with motion control and experience most of them.
That (no doubt excellent, but) technical PDF is overselling the problem somewhat, when what every dev needs to know is better represented by a friendlier summary like https://floating-point-gui.de/
No. It's been quite some time since I realized that all language is a pidgin used to translate between individuals' unique lived experiences and points of reference. And find communication much more fluid and less confrontational when the focus is on shared meaning rather than perfect word choice. Especially when working with non-native speakers, but also just people in general. Stephen Fry captures the feeling: https://www.youtube.com/watch?v=Ovi7uQbtKas
When TZubiri made their original comment, I understood they were speaking about some or all of the issues outlined in the paper I linked. If you didn't, that's ok. If you think the referenced paper missed something, it's OK to add that.
These kinds of comments tend to happen when you are assuming a shared meaning that isn't so shared. Especially in this context, it's a good idea to be precise in your terminology when there is a commonly used vocabulary for talking about it. Especially when it comes to misconceptions about floating point, which is often treated as 'random' when it generally isn't.
Sometimes specificity matters. Usually when looking directly at some specific piece of code. In this conversation, the intended meaning was clear. Folks just love being the pedant and the brinksmanship which comes with it. There's a propensity to derail otherwise useful conversations into discussions like this rather than dig deeper into what people meant, and were trying to communicate.
If everyone goes around mowing lawns for each other, the economy is richer in lawn mowing at the expense of all the other things that would have been funded had everyone mowed their own lawns and purchased different services instead.
I am confused with this, if "everyone mowed their own lawns" then the net result will be exactly the same, everyone will be busy the same and not poorer, just without money movement.
This is not the same. If everyone wants mowed lawns, and everyone is busy working on that, there is no opportunity cost, everyone is working on their top priorities. The broken window fallacy is a fallacy because the headline gdp figure doesn't account for the destruction of the window which cancels out the benefit. In the grass mowing analogy nothing has been destroyed, useful and priority work has been done all around.
1. A permutation that does as much as possible of the actual bit-mixing and
2. The simplest compression rule possible, though combining can be tricky.
Good permutations are much easier to design than good hashes, and one of the main ways hash functions are used is consuming integers smaller than the state space. May as well take advantage of provably ideal behavior.
My middle school had a reading program where the monthly winner would get a book of their choice from the scholastic catalog. This was a very enticing prize to a poor, voracious reader like me.
After winning my way through the Hobbit/LOTR series, they changed the rules specifically for me to require oral exams on each of the books I was reading, and later book reports. I was eventually banned completely so there could be an actual competition.
Reminds me of Michelson's (of Michelson-Morley) famous statement in 1900 that all of physics had essentially already been discovered, so the only remaining work was to apply what was known to new experiments. Similar statements were made about chemistry after Mendeleev and history after the cold war.
I didn't claim you wouldn't discover things in pure math that would decades later turn out to be useful in other fields. I'm sure you would. My claim was that the ROI is lower compared to applied math, which matters when funding is limited.
There's deprivation of future value and deprivation of present value. The owner can continue to sell a digital apple, but it's not clear to me that "theft" inherently deprives the owner of future value. For example, I could steal your car for a night of joyriding and return it unharmed the next day. You might reasonably say that a theft occurred even though you retain the car afterwards. Note that I'm aware this example isn't perfectly analogous to digital goods.
You grow an apple. You plant the seeds and scare off pests for like 12 years and then harvest your first apple.
Apples are scarce resources. You are very much entitled to compensation when selling the apple.
However, despite all the labor, if you could magically copy the apple, feeding the entire human race an infinite amount of apples forever, you would be pretty much morally obligated to do it. Like you wouldnt be faulted for pursuing the value of your labor once, but the idea that you would restrict the flow of apples, arbitrarily, just to try and get the value of your labor for each of an infinite amount of apples is absurd. To further imply that someone was stealing if they stole a copied apple is laughable. Really, any of your infinite apples that rots away in storage rather than finding its way to a hungry person is stealing from the hungry person.
What matters, is scarcity. Non scarce resources really shouldnt follow the same rules as scarce ones.
To add to this in some other direction, I have often thought that the people who have been "stolen" from... should just "get even" by making use of the LLM for their own profit in some way
They helped build an incredible tool that they should benefit from, so go ahead and reap the benefits, right?
Is that a serious question? Many people would consider Potala palace, the Milan Duomo, Neuschwanstein, etc more beautiful than anything designed by Zaha Hadid and Gehry. Myself included, even as someone who likes their work. Limiting ourselves only to the category of modern event centers, I'd still put the Harbin Grand Theater ahead of it.
There's a fair bit of overlap and people who work in both, but yes, they're distinct specialties. I wouldn't say upset though, except in the sense that grocery store clerks are upset by "if it doesn't scan it's free" jokes.
This hack has some interesting advantages. Float to integer is still only a few cycles, the masking is one line of libm functions, you get better (and dynamically selectable!) precision, it has gradual underflow and overflow, you can write numeric code like usual, and normalization is automatic.
reply