—

It’s Friday evening. Me and my coworker Jazz are sitting at my desk. Our secure telephony client written in Objective-C is failing to initiating calls and we don’t know why. We catch some minor things, until Jazz notices a specific reproducible problem: the client is sending a JSON-encoded session id that’s slightly off. It’s almost correct, except for the last few digits.

I figure the bug has to be in the JSON serialization code. It doesn’t matter that I’ve previously stepped over the serialization code line by line in the debugger, and confirmed its correctness by checking the raw values against the corresponding JSON. It doesn’t matter that we’re not actually doing anything fancy, just calling the built-in JSON methods. There’s simply nothing else that touches the session id: we never generate ids client side, and we never do arithmetic on them.

I must be missing something stupid.

Serial Confusion

I decide to start from the simplest possible parsing code that works, using only actual data received from the server, and then slowly add details until the bug appears. This is a good approach to take when a bug renews your fear that computer gnomes might really exist.

I make a tiny test that simply has to work:

NSString* json = @"{\"sessionId\":6032314514195021674}";
    
NSData* jsonData = [json dataUsingEncoding:NSUTF8StringEncoding];
NSError* err;
NSDictionary* obj = [NSJSONSerialization JSONObjectWithData:jsonData options:0 error:&err];
    
NSNumber* sessionId = [obj objectForKey:@"sessionId"];
test([sessionId isEqualToNumber:@6032314514195021674]);
PASS

See? It works. The parsed session id equals the correct value. Good. It would have been really inconvenient if that test failed.

I need to add a tiny detail now. We actually store the int64_t value of the NSNumber instead of the NSNumber itself. That value is accessed by using ‘longLongValue’.

I test that longLongValue also returns the right value:

...

test([sessionId isEqualToNumber:@6032314514195021674]);
PASS
test([sessionId longLongValue] == 6032314514195021674);
FAIL: 6032314514195021824 != 6032314514195021674

Wait… what? The wrong long long came out.

Long longs are guaranteed to have at least 64 bits, and this particular session id should fit into a 64 bit integer, so there should be no rounding or other tom-foolery. Maybe there’s some issue with the longLongValue method, or storing such large values in an NSNumber?

I test if an NSNumber literal has the same issue:

...

test([sessionId isEqualToNumber:@6032314514195021674]);
PASS
test([sessionId longLongValue] == 6032314514195021674);
FAIL: 6032314514195021824 != 6032314514195021674
test([@6032314514195021674 longLongValue] == 6032314514195021674);
PASS

WHAT? The long long I pulled out of the parsed NSNumber is not equal to the value I pulled out of an equal literal NSNumber.

I’m really confused. Equal things are supposed to give equal results when run through a function. That’s like… a fundamental part of being equal.

Maybe I’m supposed to be accessing this value in a different way? I start trying things. Using unsignedLongLongValue… same problem. Using decimalValue… error? Weird. Using description…:

...

test([sessionId isEqualToNumber:@6032314514195021674]);
PASS
test([sessionId longLongValue] == 6032314514195021674);
FAIL: 6032314514195021824 != 6032314514195021674
test([@6032314514195021674 longLongValue] == 6032314514195021674);
PASS
test([[sessionId description] longLongValue] == 6032314514195021674);
PASS

YOU HAVE GOT TO BE KIDDING. The description of the session id has more precision, with respect to the result of longLongValue, than the session id itself!

This is pretty weird, but at least it explains why I thought the code was correct when I stepped through it. When I logged values or looked at them in the debugger, I was seeing their descriptions. I assumed that the descriptions accurately reflected the values. Apparently that was a bad idea. Still… what’s going on under the hood, to make all of this happen?

Seeing Double

If I was coding in Javascript (the language JSON is derived from), the reason for the last few digits changing would be obvious: all numbers are doubles in JavaScript. Doubles can’t represent all 64 bit integers. My test session id (6032314514195021674) is one of those unrepresentable-as-double integers. Casting it to a double and back rounds it to, you guessed it, the extracted long long value (6032314514195021824).

Of course, I’m coding in Objective-C, not JavaScript. I am parsing data in JavaScript Object Notation, but the RFC that describes the JSON format makes no mention of interpreting numbers as doubles. The closest thing I found is in Section 4: “An implementation may set limits on the range of numbers.”. Whoever implemented the JSON parsing in Objective-C must have chosen to save time by parsing all numbers as if they were doubles, even if a particular value couldn’t be represented as a double but could be represented by another common numeric type in Objective-C.

The decision to treat all numbers as doubles is questionable, and inconvenient for me since the python and Java code I’m supposed to be interacting with both handle 64-bit numbers in JSON just fine, but it’s not wrong.

What is wrong is saying the two NSNumbers are equal when they don’t contain the same value.

Questionable Equality

Remember from earlier, that the parsed session id was considered equal to an NSNumber literal that ended up having a different longLongValue. My best guess as to why this happens is that a comparison between an NSNumber containing a long long and an NSNumber containing a double just secretly throws away precision on the long long before doing the comparison, by casting to double. I decided to test this, to be sure:

NSNumber* d = @6032314514195021674.0; // double
NSNumber* n = @6032314514195021674; // long long

test([d isEqualToNumber:n]);
PASS
test([d longLongValue] == [n longLongValue]);
FAIL: 6032314514195021824 != 6032314514195021674
test([[d description] isEqualToString:[n description]]);
FAIL: "6.032314514195022e+18" != "6032314514195021674"

Ugh.

Throwing away precision before checking for equality is a highly questionable decision on Apple’s part. It’s as if 0.3 == 0 evaluated to true because (int)0.3 == (int)0. Comparing a float to an integer might be hard to do correctly, but that doesn’t mean you can just pretend a long long will fit into a double!

Side note: In many languages, the expression 6032314514195021672 == 6032314514195021674.0 will return true, because 64-bit integers are automatically promoted to doubles. I think that implicitly promoting 64-bit integers to doubles is a mistake (because it loses precision), but it’s worse with NSNumber because the types aren’t available. A static verifier can detect comparing an int64_t to a double very easily, but if you tried to do the same thing with NSNumbers you’d be swamped by false positives because it’s difficult to prove that an NSNumber can’t be a double or an int64_t.

Moving on from the throwing-away-precision-when-comparing issue, there’s one interesting difference highlighted by the above test. The NSNumber created from the double literal has a description whose precision matches its value. That contradicts what I saw with the NSNumber parsed from JSON.

Other Numbers

It’s important to understand that objective C has interfaces, instead of classes. Even basic types like NSNumber may have multiple implementations. There might be an NSNumber for doubles, an NSNumber for ints, and maybe lots of other specialized ones. Not that you should have to care, because they all conveniently expose the same interface: NSNumber.

With that in mind, I think the NSNumber returned from the JSON parser is a specialized NSNumber. One modified to keep a copy of the JSON it was parsed from as its description. This would be useful for uses cases like modifying JSON without losing information in the parts you didn’t touch.

Putting it all together, I have a plausible/questionable reason for each part to work the way it does and I understand how they combined to trick me for so long:

• Numbers parsed from JSON are all treated as having double-precision; probably because doing that saved on expensive development time.
• Numbers parsed from JSON use the text they were parsed from as their description even if that description has extra precision; probably to allow better round-tripping of data.
• NSNumbers that contain long longs have their values rounded when being compared to NSNumbers containing doubles; probably because it was easier than doing it correctly, and that’s what == does in C.
• The above three combined mean a number parsed from JSON will acts as if it has full precision, when printed out or compared to the expected value, but it really doesn’t.

Tricky. Complicated tricky.

Summary

Objective-C lies about how much precision NSNumbers parsed from JSON have. If you print one out, you see the exact integer value that was parsed. If you compare one to an NSNumber containing the expected value, they’re equal. If you actually try to access the integer value, it’s been rounded to the nearest representable double.

You can work around the issue by using the numeric value of the description:

// broken:
int64_t roundedValue = [lyingValueParsedFromJson longLongValue]
// "fixed":
int64_t actualValue = [[lyingValueParsedFromJson description] longLongValue]

Make sure you leave a comment explaining that workaround, or maintainers will face-palm for the wrong reason.

—

Discuss on Reddit

—

Tinkering

Years ago, I was working on a hobby project I later called Tinker. Tinker was a game hosting bot for WarCraft 3.

Hostbots are useful for a variety of reasons I won’t go into, but mostly I enjoyed Tinker as a technical challenge: a way to experiment with small amounts of reverse engineering, to explore programming techniques (e.g. actor model, strategy pattern, parser combinators), and to seriously try out some code verification tools (e.g. Code Contracts, Code Analysis, Pex, CHESS).

I did most of my reverse engineering by analyzing the information exchanged between Battle.net and the WarCraft 3 client. I would start WarCraft 3 and Wireshark, perform an action in the game, see what information was sent, figure out the structure of the information, and program the bot to mimic that structure.

Because I was inferring the structure of packets by observation, I’d often miss small details. I’d think some value was always 0, but it turned out there was a corner case where it was supposed to be 1, or whatever.

Typically, the consequence for getting a detail wrong was all of the players immediately disconnecting from the bot’s game, or the bot being immediately disconnected by Battle.net. An annoying but ultimately minor consequence, in the scheme of things. Typically.

Making a Game and a Bug

As a player, creating a custom game in WarCraft 3 is relatively simple: you choose a map and a game name then click “Create Game”, for the most part.

As a programmer, things aren’t so simple. You need to know the type of packet used to create a game, where the game name goes in that packet, how the game name is encoded (null-terminated UTF8), how maps are identified (path relative to program folder and a few sometimes-quirky checksums), where they go in the packet, what game settings there are, how they are encoded (bit flags), if there’s any sort of trickery to avoid zeroes because all these things are being shoved into a null-terminated string (yes), and etc. Remember: get a detail wrong, and your user gets disconnected instead of making a game.

Obviously, as part of writing a game hosting bot, I had to write code to generate ‘create game’ packets (a lot of the work had already been done). However, in doing so, I missed one important caveat: in Warcraft, the game name text box stops taking input after 31 characters. In fact, game names longer than 31 characters are considered invalid.

Whatever. No big deal, right? Someone will try to make a game with a long name, and their bot will get disconnected. They’ll contact me, give me the logs, and I’ll figure out the problem. Typical bug fix. Annoying for the user, but nothing serious.

No Games for Anyone

Things weren’t so typical, this time. I had no idea anything was wrong until I received a private message:

Urgent: HostBot

gamename = “0123456789012345678901234567890″ // (legal)
gamename2 = “01234567890123456789012345678901″ // (illegal)

You need to implement a max gamename length to your HostBot program NOW. If a game is hosted on a server with a name that is longer than the max game length imposed by WC3, everyone who views the Custom Game list on that server gets instant disconnection [emphasis mine].

It appears the max gamename length is 31 characters.

I love your HostBot program, don’t get me wrong but… your software is the only freely available software I know of that can create this problem, and its been happening frequently (once every couple of days on USEast, the custom game list goes down for hours at a time).

Its pretty hard to believe that Blizzard have left such a problem in Bnet, but their software was never setup to deal with the freedom your HostBot currently gives.

Thanks!

Oh Sh-…..oot. Tinker allowed too-long game names, and the Battle.net servers were accepting them! The servers would then broadcast the invalid names to any WarCraft 3 client that asked for the custom games list, causing the client to disconnect from Battle.net. Users of my bot were accidentally performing denial-of-service attacks on the custom games list and taking it down for the entire realm for hours at a time!

Hundreds, maybe thousands, of people were being actively inconvenienced by a mistake I made. I fixed the bug as quickly as possible. I informed Varlock/hogantp, maker of the better-known GHost. I informed Blizzard.

I was a bit worried blizzard would just ignore the issue, becauset of my experience with them chronically ignoring suggestions made by the mapping community. However, a few days later, sending a too-long game name would correctly disconnect the bot instead of everyone else. Maybe Blizzard noticed the problem on their own.

Summary

Users of Tinker were accidentally preventing everyone from playing custom WarCraft 3 games, because I didn’t realize (and Blizzard half-forgot) that game names were required to have fewer than 32 characters.

My bug performed a (sort of) distributed denial of service attack on (a subset of) Battle.net. My bad.

(Fun fact: being a programmer lets you make bigger mistakes. I think my mistake inconvenienced on the order of ten thousand people. That’s a lot, but laughably small on the scale of things programmers have messed up.)

—

Discuss on Hacker News, Reddit

—

“Type operators that collapse when you nest them inside of themselves”, along with some conditions to actually work as you would expect, precisely describe the idea of monad.

[...] This is the textbook definition of a monad.

monads, anyone?

So the article and discussion here bring up a key question: what are the monads such that m (m a) and m a are isomorphic?

Given the mix of confusion and interest, I want to clear up how monads relate to collapsing types.

Different

Collapsing types aren’t the same thing as monads, and neither is a subset of the other. They are attributes that can apply independently.

A collapsing type is a type that satisfies the property that nesting it inside itself has no effect. A must act identically to a under all circumstances, for to be a collapsing type.

A monad is a type that supports three operations: wrapping, transforming, and flattening (more typically called , , and respectively). The operations must also satisfy a few work-in-the-obvious-way-please rules.

If you tell me that is a collapsing type, I still don’t know if the type supports the operations necessary to make it a monad. Collapsing types technically support flattening, in that a proper method doesn’t have to make any changes. However, collapsing types are not required to support wrapping or transforming.

Conversely, if you tell me that is a monadic type, I still don’t know if it satisfies the condition needed to qualify it as a collapsing type. Monads are not required to have operations that change nothing.

Examples

There are non-collapsing non-monadic types, collapsing non-monadic types, non-collapsing monadic types, and collapsing monadic types. Just to really drive home that “collapsing” is separate from “monadic”, lets go over an example of each.

An example of a collapsing monadic type is , from last week’s post.

An example of a non-collapsing monadic type is . You can wrap anything into a list, transform the items in a list, and flatten lists of lists but a list of lists of integers is not the same as a list of integers.

An example of a collapsing non-monadic type is a type representing an unknown value stored on a remote server, where the details of the server potentially using a further-remote server are hidden (meaning you treat a exactly like a ). If the server doesn’t allow you to apply arbitrary operations to remote values, or create arbitrary remote values, then is not a monad.

An example of a non-collapsing non-monadic type is . It’s not a collapsible type, because a matrix of matrices of booleans is distinct from a matrix of booleans. It’s not a monad unless you specify a flattening operation, but you’ll find there aren’t any nice ones to pick because you can’t preserve the number of values. For example, properly flattening would require that seven be a square number (it’s not).

Side note: Technically you could make SquareMatrix into a monad by specifying a poor flattening operator, and so you might be tempted to say SquareMatrix is a monad. Don’t do it. Under that view, every type ever is a monad. Given a type T, you can make it a useless monad by picking a value v of type T and defining join, fmap and return to all unconditionally return v.

Similar

Monads and collapsing types are different, but still related.

In practice, collapsing types you encounter will probably be monads. Mainly because most of them will be related to eventual results (e.g. promises/A+ in JavaScript, Twisted Python in Python, ppltasks for windows store apps in C++).

In theory, you can start from any monad (which may not be a collapsing type) and derive a related collapsing+monadic type by making the application of automatic. It’s what I did with last week. All the methods on did internal checks to see if flattening was necessary or not, to make a act exactly like a . This derivation makes more practical sense with some types (, , , ) than it does with others (, , , ).

Warning: a potential issue with the above derivation is the creation of self-referencing flattening cycles, where you end up with a type nested inside itself infinitely many times and keep trying to unwrap it. For example, adding the following three lines to my collapsing future’s python fiddle causes a flattening cycle:

cycle = Future()
cycle.trySetResult(cycle)
cycle.continueWith(out) #never halts

It’s probably a good idea to have runtime checks for this issue. Compile-time checks would be even better, but beware accidentally solving the halting problem.

Summary

Collapsing types are not the same thing as monads.

You can transform any monadic type into a collapsing monadic type.

Next week: less type theory.

—

Discuss on Reddit

—

Collapsing the Eventual Eventual Future

Suppose you have a Future: a representation of a value that may not be computed yet but will eventually be ready. Further suppose the eventual value of that future (call it ) is itself a future (call it ), and that the eventual value of is the integer . Said another way, is eventually eventually . Notice that “eventually” was repeated twice.

An interesting property of the word “eventually” is that repeating it twice doesn’t seem to change the literal meaning of sentences. Saying something will eventually eventually happen is the same as saying it will just eventually happen. Doubly-eventual is the same as singly-eventual. However, your typical implementation of a type will not have this “doubly is the same as singly” property. You’ll find that that , even though is eventually eventually , because is a instead of an .

Can we implement a variant of , a , that does respect that singly-eventual should be the same as doubly-eventual? This would be useful because it would mean we wouldn’t have to track the level of nesting in order to write correct code (which is particularly useful in dynamic languages and languages without generics, where the compiler won’t catch nesting-level mistakes).

Side note: If you think of as a function that takes a type and returns a type, then calling a collapsing type is equivalent to saying its function is idempotent.

If we try to implement in C# or in Java, we’ll run into a bit of trouble when we try to specify the type of the value in a . It’s a , unless is a in which case it’s a , unless is… you get the idea. Their type systems can’t express these sorts of conditions, although more flexible type systems can (I think you can do it with templates in C++, for example), so we’re forced to give up on collapsing or give up on knowing the type of value coming out of a future.

In a dynamic language, or in a language without generics, this trade-off doesn’t exist. The language designers have made the choice for us: the type of value coming out of a future can’t be specified. This makes implementations in such languages feel much more natural.

Python Implementation

I implemented a collapsing future type in python, which you can play with on PythonFiddle. It works by checking the types of results at runtime, and flattening them when they’re also futures. Here is the type:

### An eventual value that automatically flattens doubly-eventual results
class Future:
    ### Inits a new not-yet-specified future value
    def __init__(self):
        self.__continuations = []
        
    ### Registers a continuation to run when the flattened future value is ready
    ### Returns the continuation's future result
    def continueWith(self, continuation):
        # remember continuation, if result not provided yet
        if self.__continuations != None:
            r = Future()
            self.__continuations.append((continuation, r))
            return r
        
        # automatic flattening
        if isinstance(self.__result, Future):
            return self.__result.continueWith(continuation)

        # run continuation inline when already finished        
        r = Future()
        v = continuation(self.__result)
        r.trySetResult(v)
        return r
    
    ### Sets this future's value, if it has not yet been set.
    ### Causes all registered continuations to be run or handed off.
    def trySetResult(self, result):
        # already set?
        cs = self.__continuations
        if cs == None: return False;
        
        # state transition
        self.__continuations = None
        self.__result = result
        
        # perform or hand-off registered continuations
        for e in cs:
            continuation, future = e
            r = self.continueWith(continuation);
            future.trySetResult(r)
        
        return True

If you read closely, you’ll notice that the above (not particularly efficient) implementation is itself (ab)using the fact that is a collapsing type. For example, when a future’s result is being set to a a future, the results of continuations are set directly to the results of handed-off continuations despite them having a different level of nesting.

Here’s some sample code that uses the above future type, demonstrating that you don’t have to care about the level of nesting:

f1 = Future()
f2 = Future()
f3 = f1.continueWith(lambda v1: f2.continueWith(lambda v2: v1 + v2))

def out(x): print x
f1.continueWith(lambda x: out("f1's result = " + x))
f2.continueWith(lambda x: out("f2's result = " + x))
f3.continueWith(lambda x: out("f3's result = " + x))

f1.trySetResult(f2)
f2.trySetResult('Hello')

# outputs: 
# f2's result = Hello
# f3's result = HelloHello
# f1's result = Hello

In the above code, has type because its result is set to a string. has type because its result is set to be . has type , for reasons related to continueWith returning a when given a function that returns a .

Wait, why are we doing this complicated analysis? Because of the collapsing property, it’s dead obvious that they all have type . The fact that ‘s result prints out as "HelloHello", instead of <__main__.Future instance at ...>, confirms it.

This is a level beyond having the compiler detect that you’ve made a nesting-level mistake. The mistake has been made fundamentally impossible, at least with respect to our future type on its own.

Mixing With Others

There are other types that can benefit from being made collapsible. For example, the “MayFail” and “MayBeCancelled” types I mentioned last week would work well as collapsing types, because you very rarely care “at what level” a failure or cancellation occurred. On the other hand, “List” would be a terrible collapsing type because you so often need to have a lists of lists. Another terrible candidate is “Maybe”, because you usually do care about “at what level” a value has been omitted.

One potential obstacle to having lots of collapsing types is mixing. A won’t automatically collapse into a , and so mixing may re-introduce the need to care about nesting level. This is particularly bad in the case of futures because they often represent operations that might fail, and so mixing will be endemic. Maybe it would be possible to have some sort of cross-type collapsing that applies in appropriate places… I really just don’t know yet.

Summary

A collapsing type transparently flattens itself when nested inside of itself. A will act exactly identical to a .

Collapsing types are easy to use, because they make nesting-level mistakes impossible. Collapsing types are hard to represent, because their type can’t be specified in the type systems of most popular languages.

—

Discuss on Reddit

—

In this post, I’ll explore this issue from the perspective of eventual results / promises / futures.

Representing Eventual Results

The standard promise/future/eventual-value type in .Net is called Task<T>.

Like all decent ‘eventual result’ types, Task<T> supports or can support all the operations needed to qualify as a monad (wrapping, transforming, flattening):

// wrapping
Task<int> wrapped = Task.FromResult(10);

Task<bool> wrapped2 = Task.FromResult(true);

// transforming
Task<IPAddress> transformed = from hostEntry in Dns.GetHostEntryAsync("example.com")
                              select hostEntry.AddressList.First();

Task<Task<TcpClient>> transformed2 = from address in projected
                                     select Tcp.ConnectAsync(address, 80);

// flattening
Task<TcpClient> flattened = transformed2.Unwrap();

However, Task<T> is more complicated than strictly necessary. A simplest-possible eventual result, a MinimalTask<T>, would only be able to represent eventually succeeding with a value of type T. A Task<T>, on the other hand, can succeed with a value of type T, or fail with an exception, or even end up cancelled with no associated value. When you ask for a Task<T> you’re actually asking for a MinimalTask<MayFailWithException<MayBeCancelled<T>>>!

Of course, there are good practical reasons for Task<T> to be able to represent failure and cancellation. If you try to run a function asynchronously, and it throws an exception, it’s downright useful to have a place to put that exception. Most operations can fail. Most asynchronous operations can be cancelled. It makes sense to support those scenarios.

Notice that .Net having exceptions just leaked into how we represent eventual results.

All tasks can fail or be cancelled. That means everything that works with tasks must handle cancellation and faulting in an appropriate way. The flexibility is useful, but when it isn’t needed it unnecessarily complicates things and introduces opportunities for bugs. (Note similarities to the downsides of forcing all references to allow a null value.)

A good example of the trade-offs due to cancellable faultable tasks is the design of Task.WhenAll.

Aggregating Eventual Results

Task.WhenAll takes a sequence of tasks and returns a task that completes when all of the tasks have completed. More exactly, once all of the tasks have completed, they are combined as follows. If any of the tasks failed, the resulting task fails with an aggregate exception containing all of the failures. Otherwise, if any of the tasks were cancelled, the resulting task ends up cancelled. Otherwise the resulting task succeeds with an array of the tasks’ values.

Notice that, if tasks couldn’t represent exceptions, the above description would be a lot simpler. It wouldn’t discuss how to aggregate exceptions, it wouldn’t note that exceptions take priority over cancellations, etc. The majority of the method’s specification is made up of details that aren’t directly related to eventual-ness, and could exist elsewhere.

A hypothetical WhenAll method for a hypothetical MinimalTask<T> type would be simpler than Task.WhenAll. Since there’s no exceptions or cancellations, there’s no need to specify the details of combining exceptions, whether exceptions take priority over cancellation, and etc. MinimalTask.WhenAll “flips” the order of many-ness and eventual-ness in the obvious way, and that’s it.

Can we implement Task.WhenAll in terms of the hypothetical MinimalTask.WhenAll? Keeping in mind that a Task<T> is essentially a MinimalTask<MayFail<May<T>>>, it’s actually pretty easy. All of the details of how to aggregate should already be in the types designed to represent potential-failure and potential-cancellation, so the method ends up being downright simple:

MinimalTask<MayFail<May<T[]>>> WhenAllMay(this IEnumerable<MinimalTask<MayFail<May<T>>>> tasks) {
    return from v1 in MinimalTask.WhenAll(tasks) // v1: MayFail<May<T>>[]
           from v2 in MayFail.WhenAll(v1)        // v2: May<T>[]
           from v3 in May.WhenAll(v2)            // v3: T[]
           select v3;                            // result: T[] in May in MayFail in MinimalTask
}

(Interesting fact: actually this method is in a sense not even necessary because all WhenAll methods can be implemented with the same code.)

By separating eventual-ness from may-cancel-ness and may-fail-ness, we avoid repeating the details of how to aggregate exceptions and cancellations.

Wrapping Up

The technical term for a type that represents ‘is a result of type T or else a failure’ is “Error Monad” (it’s a monad because you can wrap, transform, and flatten may-have-failed-ness).

[Aside: I can't believe the best 'error monad' link I could find was technical Haskell documentation. If anyone has a more informative link, I would appreciate it.] (old link, new link suggested by DR6)

Unfortunately, although it may be aesthetically pleasing and proper functional style to separate may-fail-ness from eventual-ness, I’m not sure if it’s a good idea or not in imperative languages with exceptions. The problem is that they already have a way to represent failure, and the easiest way to fight your language is to add second way to do something included “in the box”.

Summary

Because .Net has exceptions, Task<T> is forced to represent exceptional results. Because Task<T> can represent exceptional results, async methods are often more complicated than necessary (making them harder to work with, functionally). Separating the ‘might fail’ part out of Task<T> and into its own type is one way to avoid the complications when they aren’t necessary. However, because .Net has exceptions, doing so is potentially bad form.

That’s why I consider tasks to be an example of what I mean when I say exceptions mix poorly with a functional style. They add complications to otherwise-simple functional types.

—

Discuss on Reddit

—

Returning readers might see the issue right away. Don’t worry, it’s got nothing to do with unicode.

Flunf?

Why the flunf does the following C# code print ‘False’?

var flunf = 98765432f;

var sortedList = new List<double> {
    flunf + 1, 
    flunf + 3, 
};
var valueToInsert = flunf + 2;
var indexOfInsert = sortedList.TakeWhile(e => e < flunf + 2).Count();
sortedList.Insert(indexOfInsert, valueToInsert);

var stayedSorted = sortedList.SequenceEqual(sortedList.OrderBy(e => e));
Console.WriteLine(stayedSorted); // prints 'False'

(You should check your solution against the ‘likely mistake’ entry below.)

Hints

Hint: The fact that flunf is a float matters, but how it matters is the real puzzle.

Hint: sortedList is [98765433, 98765432, 98765435] in the end.

Hint: The type of float+int is float.

Spoiler: The preceeding hint is correct, but very misleading.

Likely Mistake: If limited precision was the only issue, everything would round to flunf and be trivially sorted.

Disclaimer: It is possible that, in the future or on other machines, the list will actually end up sorted and ‘True’ will be printed. In that case, the puzzle is why and under what conditions could the list end up not sorted? I assure you that the code prints ‘False’ on my machine.
—

Discuss on Reddit

—

So, this week, I’m going to explain how to solve millionaire-style problems without resorting to any sort of cryptography.

The Problem: Socialist Millionaires

Two employees, Alice and Bob, want to know if they are being paid fair wages. They do the same amount of work at the same level of quality, but they suspect that their employer might be taking advantage of one of them.

The easiest way to determine if they make the same amount of money per hour is for both to say “I make X dollars per hour”. Unfortunately, although Alice and Bob trust each other, they are very private about their income. They consider it a huge social faux pas to reveal how much money you make. The possibility of common knowledge of who makes more money is just… too uncomfortably awkward to bear.

Not only do Alice and Bob refuse to reveal how much money they make, to each other or a trusted third party, but they’re a bit technophobic and refuse to enter the information into a computer for fear of it being logged.

How can they find out if they’re being paid fairly, without risking an awkward situation and without using computers? With locked suggestion boxes, of course!

The Solution: Use Locked Suggestions Boxes

Suppose Alice and Bob each might be making either 10, 20, 30, or 40 $/hour. We’ll arbitrarily say that Alice makes 30$/hour and Bob makes 20$/hour.

1. Bob goes to an office supply store and buys four lockable suggestion boxes (with different matching keys). He labels the four boxes as 10$, 20$, 30$, and 40$.

   

2. Bob discards all of the keys except the key for the 20$ box (because that’s how much he makes per hour).

   

3. Bob gives the locked suggestion boxes to Alice. In private, Alice puts a slip of paper saying ‘yes’ into the 30$ box (because that’s how much she makes per hour). She puts slips of paper saying ‘no’ into the other boxes.

   

4. Alice gives the boxes back to Bob. In private, Bob uses his key to unlock the 20$ box and get the slip of paper inside.

   

5. Bob sees that the slip of paper says ‘no’, meaning Alice doesn’t make 20$/hour like he does. He tells Alice they don’t make the same amount of money.
6. Bob now knows that Alice doesn’t make 20$/hour, but hasn’t learned if she makes 10, 30, or 40 $/hour. Similarly, Alice now knows Bob doesn’t make 30$/hour, but hasn’t learned if he makes 10, 20, or 40 $/hour.

Oblivious Transfer

The technical term for what Alice and Bob did in the previous example is oblivious transfer. Alice transferred many messages to Bob, but is oblivious to which single message Bob received. Alice sent an answer for every possible amount of money Bob might make, but Bob only received the answer corresponding to how much money he actually makes.

Note that there’s a certain amount of trust required for this to work. If Bob is going to lie and pretend he makes 30$/hour, there’s no way to detect the problem. There are also other ways to cheat, but they can mostly be detected or prevented by adding complications to the solution. For example, Bob can show Alice the slip he pulled out, to prove he’s not just lying about the outcome. I don’t want to dig too deep into the details of what types of cheating you can prevent… suffice it to say that it goes very deep.

If Bob and Alice were using computers, instead of physical items, they would have used public key cryptography instead of locked suggestions boxes. This makes the computer protocols harder to understand, but removes a lot of corner cases inherent in the physical protocol. It’s much easier to prevent Alice from secretly marking the slips of paper saying ‘no’, so she can determine which box Bob opened, for example.

Summary

You can perform oblivious transfers without computers, using locked suggestions boxes.

The ability to do oblivious transfers is the basis for performing any secure multi-party computation.

—

Discuss on Reddit, Hacker News

—

This post is… basically just a list of computer science things I think are cool.

Physical

I remember a time when I was very confused about how computers worked. I knew I could type in a program, and the computer would run it, but how that happened was a big mystery. I could write a program to interpret programs, but… how do I run the interpreter? It can’t be interpreters all the way down!

When I learned how to make an adder out of logic gates, and realized I could pull the same sort of trick all the way up… I imagine that’s how Archimedes felt as he ran through the streets naked after realizing he could measure volume via water displacement.

It sounds a bit silly to say it, but did you know computers run on physics?! Amazing!

Coloring Complexity

Suppose you’re given a map. Your job is to determine if, with a particular number of colors, you can color in all the areas without giving any bordering areas the same color. You don’t have to actually provide such a coloring, just determine if there is one.

How difficult do you think it is to solve this problem? It actually depends on the number of colors.

With 0 or 1 colors, the problem is trivial. Only graphs without nodes are 0-colorable, and only graphs without edges are 1-colorable.

With 2 colors, the problem is easy. You can solve it in linear time. As soon as you pick the color of one area, you’re forced to a single possibility for adjacent areas. Just keep filling in areas as you’re forced to, until you either finish or find a contradiction.

With 3 colors, the problem is very hard. In fact it’s NP-Complete, meaning an algorithm that solves any instance in polynomial time could be repurposed to say… be the first to find inputs that collide when hashed with SHA-1.

With 4 colors, the difficulty changes again. Complicated details aside, you can use the following algorithm to solve the problem:

bool IsFourColorable(PlanarGraph g) {
    return true;
}

Needless to say, planar 4-colorability is a bit easier than planar 3-colorability.

Over-constrained problems are easy, but so are under-constrained problems. Changing tiny details of problems can massively change how easy they are to solve.

Proofs without Proofs

Suppose you meet an alien with access to absurd amounts of computational power. The alien is cooperative, but you’re not sure if it’s trustworthy. Can the alien help you solve hard computational problems, even though you don’t trust it and it can probably track everything you do? Can Arthur extract useful work out of a tricky Merlin?

Welcome to the wonderful world of interactive proofs.

Want to convince me that you have solved a problem or know some information (e.g. a password), without giving me the solution or the information? You can do it with an interactive zero knowledge proof. For example, most cryptographic voting systems involve voting machines zero-knowledge-proving that they are functioning correctly at every step.

Want to search throughall the different possibilities that could be reached in exponential time, to see if one meets a criteria? All you need is two isolated provers instead of one. No, seriously, MIP (problems solvable in polynomial time with multiple unbounded interactive provers) is equivalent to NEXPTIME (problems that can be solved in exponential time, with access to a ‘branch both ways’ instruction). This is not quite as practical as zero knowledge proofs, but NEXPTIME is so gigantic that I don’t care.

In computer science, the idea of a mathematical proof is one special case amongst many strategies to convince verifiers that something is true.

Secure Multiparty Computation

A common intuition in real time strategy games is that map hacks are unavoidable. The game needs to know where enemy units are in order to determine if they can be seen, and a map hack is just a matter of displaying instead of hiding the information… right? Wrong.

Secure multiparty computation (SMPC) allows mutually distrusting computers to interact in a way that simulates a trusted server. In the case of an RTS game, the simulated server would compute the answers to questions like “What units can see each other?” and then tell the players what they can see. This can be done without any of the players’ computers learning where hidden enemy units are.

Unfortunately, SMPC is quite expensive and requires back-and-forth interaction. It’s not practical to run it over hundreds of unit positions a hundred times a second. It’s more applicable to turn based games like civilizations or poker than to real time strategy games.

Still, the fact that having a real trusted third parties is unnecessary (it’s more like… an optimization) is very cool.

Halt

You’ve probably head of the halting problem. Basically, it’s a more concrete version of Gödel’s incompleteness theorem: instead of proving xor disproving every statement, you want to find a program that determines if given programs halt.

It is the case that no program solves the halting problem. This is quite possibly the most actively counter-intuitive thing I know.

For example, you can write a (partial) halting solver that works by enumerating all mathematical proofs until a proof that the program in question halts is found or a proof that the program in question doesn’t halt is found. The only way such a solver can fail is if there are “unfathomable” programs that run forever, with no possible proof even in principle that they run forever. (Alternatively: you used an inconsistent proof system.)

Unfathomable programs exist.

The idea of unfathomable programs breaks my brain. Again and again, I end up with the same confused questions: if there’s no proof the machine doesn’t halt, in what sense does the fact that it doesn’t halt even exist? How does it make any sense to require an infinitely large proof about any aspect of a machine with a finite-sized specification?

Even worse, unfathomable programs are impossible to find. Confirming you had a no-proof-but-runs-forever program would involve proving it ran forever and then proving you didn’t just prove that last thing.

Basically, the halting problem can be used to produce the feeling of confusion-about-reality-itself on demand. It’s just that mind blowing to me.

Summary

Computer science interesting. Craig like.

—

Discuss on Reddit

—

On Tuesday the day began with a prepared presentation ‘Lenses: A comfortable way to manage shared state’. It covered in depth a thought that Craig touched on in a previous blog post. Slides are available here and generic implementation + sample code is here.

After the talk I was able to meet each of the impressive teams working on great mobile games. I was especially impressed with Double Stallion’s Big Action Mega Fight (BAMF for short; Hmmmm…) on the technical side of things. I enjoyed excellent design conversation with Lightning Rod Games’ Steven Smith and Imaginary Games‘ Chris Powell; both multi-talented developer/designer/bizdudes. Andrew at Miscellaneum Studios and I talked at length about cross platform GUI, Continuous Integration and Unity’s Script Execution Order for the camera in their title The Firemasters. Finally I was able to play through Pixel Crucible’s work in progress game which is draws inspiration from some of my favorite games.

Next up was Mont Royal Game Society monthly meetup where I tried my hand at Super Punch ball and had entirely too much poutine. Then on Thursday was a 5 á 7 with TAG Concordia where I learned how to play koi koi with otakus and game designers.

Friday capped off my journey with a wonderful visit to Minority Media followed by giving a talk to the Montreal Unity User Group where I demoed my (potential) talk for Unite 2013 on extending coroutines (Based on this post but made a bit more accessible). After the talk I went to a really great japanese whiskey bar on St. Laurent. and crashed with some awesome game designers.

Montreal is one hell of a game development scene that I would be proud to call home; but boy was I ready to be back in Halifax after the 2.5 week tour-de-force. Next stop, Toronto for TOJAM8 with Petar and Paul and a MIDI controller!

You realize you can generate random values by flipping coins, but a coin flip has two possible outcomes instead of six. How do you simulate a fair six sided die using only fair coin flips? What about the more general problem, of simulating an -sided die with an -sided die?

If you haven’t seen this puzzle before today, and want to solve it yourself, stop now. I will be spoiling it.

Partial Credit

There’s a well known sub-optimal solution to this problem.

First, given a die with sides, it’s trivial to simulate a die with sides by grouping rolls. A sided die can be simulated by arranging coin flips into groups of , because each group has equally likely possibilities. The reverse direction, from to , is also easy because each roll can be split evenly into sub-rolls.

Second, given a die with more sides than necessary, you can just filter out rolls that have invalid results with respect to the smaller die. Given an -sided die, we can simulate a sided die by rolling until we almost surely get a result that’s not or .

Putting these two parts together allows us to simulate a -sided die using only our -sided coin. Just flip the coin three times, pairing each of the possible outcomes with the numbers to , and try again whenever the result is or .

There. Solved. Well… except it feels wasteful to throw away those sevens and eights.

Wasted Entropy

The above solution wastes information entropy. A coin flip generates 1 bit of entropy. Three coin flips generates 3 bits of entropy. The result of the roll of a six sided die, on the other hand, gives you only bits of entropy.

Even if we get lucky, and don’t have to try again, we’re consuming bits of entropy while only producing bits when converting our coin flips into six sided die rolls. If we’re average, instead of lucky, then we’ll need an expected attempts for an expected cost of bits. We’re wasting bits of precious, delicious entropy per generated roll!

This raises the question: can we do better? Can we use that wasted entropy?

This is your second chance to stop and solve.

Entropy Optimizations

There are several optimizations that we can make, to improve our entropy efficiency.

For example, consider what happens if we map [heads,heads,heads] to and [heads,heads,tails] to . Since they both start with [heads, heads], the third coin flip has no effect on whether or not we retry. This allows us to save one coin flip when retrying, reducing our expected cost from to bits.

Another useful optimization to notice is that factors into and , and that the product of two uniformly random numbers is also uniformly random. A -sided die roll is interchangeable with a -sided die roll combined with a -sided die roll. Since we have a coin with sides, -sided die rolls are trivial to do with perfect efficiency. Then, for each -sided roll we generate, we can generate an accompanying -sided roll by using groups of coin flips (and retrying when we get a instead of , , or ). Combined with the previous optimization, this lowers our cost to bits.

A more flexible optimization is to find a power of that comes proportionally sooner after a power of . For example, we can use our coin to simulate rolling a die with sides with perfect efficiency, discard when we exceed , and generate three rolls of a -sided die per success. It takes more coin flips to get our first result, but the probability of discarding decreases significantly. Combined with our previous two optimizations, we’ve lowered our expected cost to bits.

We can find even better powers of and . The next few good powers are , and . Each gets us tantalizingly closer to perfect efficiency, and it’s not even that hard to find good pairs. Just run the following program for sufficiently long:

BigRational r = 3;
int n2 = 0;
int n3 = 1;
BigRational best = 0;
while (true) {
    while (r > 1) {
        r /= 2;
        n2 += 1;
    }
    if (r > best) {
        best = r;
        Console.WriteLine("3^{0},2^{1}", n3, n2);
    }
    while (r < 1) {
        r *= 3;
        n3 += 1;
    }
}

(Note that the BigRational type is not standard. I referenced it from the BigRationalLibrary NuGet package.)

Now for some disappointing news: you'll never find the perfect pair of powers. We can achieve % efficiency with and . We can achieve % efficiency with and . We can get arbitrarily close, but we can't achieve % efficiency. Doing so would require an impossibility: a power of three that's also a power of two. That's not going to happen. Powers of three aren't even ever even! No matter how big we make the constants, there will always be some unlikely cases where we have to discard and try again.

Give that achieving % efficiency is impossible, and given that we can get arbitrarily close to % by choosing the right constants, a person would be tempted to think we'd hit the limit. That we can't do any better. That person would be wrong.

Last chance to solve on your own.

Conservation of Entropy

Beating 'choose how close you want to be to 100% efficiency' is just a matter of not settling for a fixed amount of efficiency. One way to do this would be to compute better constants as we generated die rolls. As the algorithm ran longer and longer, it would waste less and less entropy.

However, it turns out we can do something slightly simpler: arithmetic coding. The idea is to treat the coin flips as the binary digits after the decimal point (binary point?) of a number between and . As we add more binary digits, the space where the 'complete' number can lie becomes smaller and smaller. Eventually, that space will almost certainly fall entirely within the domain covered by a single digit in base 6. We can then safely output that digit, and start waiting for the next base 6 digit to become fixed.

This idea easily generalizes to dies with arbitrary numbers of sides. The following code implements it:

///<summary>Generates rolls of a uniformly random die, based on rolls of a uniformly random die with a different number of sides.</summary>
public static IEnumerable<int> ChangeDiceSideCountFromTo(this IEnumerable<int> rollResults, 
                                                         int oldSideCount, 
                                                         int newSideCount) {
    if (rollResults == null) throw new ArgumentNullException("rollResults");
    if (oldSideCount < 2) throw new ArgumentOutOfRangeException("oldSideCount", "oldSideCount < 2");
    if (newSideCount < 2) throw new ArgumentOutOfRangeException("newSideCount", "newSideCount < 2");

    BigRational min = 0;
    BigRational len = 1;
    foreach (var roll in rollResults) {
        // the roll represents the next digit in base-[number of old sides], reducing the valid range
        len /= oldSideCount;
        min += roll * len;

        // check for digits in base-[number of new sides] becoming fixed
        while (true) {
            var minDigit = (min * newSideCount).GetWholePart();
            var maxExclusive = (min + len) * newSideCount;
            var epsilon = new BigRational(-1, maxExclusive.Denominator * newSideCount);
            var maxDigit = (maxExclusive - epsilon).GetWholePart();
            if (minDigit != maxDigit) break;

            // another digit has become fixed and can be output
            yield return (int)minDigit;

            // normalize so the range of the next digit also ranges from 0 to 1
            len *= newSideCount;
            min *= newSideCount;
            min -= minDigit;
        }
    }        
}

The above code achieves 100% efficiency, with a small caveat: delay. If you convert from an sided die to an sided die and then back to an sided die, you'll get the exact same rolls out that you put in. That proves no entropy is lost. However, the output rolls will lag the input rolls by an expected constant amount. This delay occurs because different bases have different borders between digits.

For example, binary numbers starting with 0.1010101010 may start with 0.1 or 0.2 when represented in ternary. Until you almost certainly roll outside of the repeating 101010 pattern, the next ternary digit can't be determined. On the bright side, when you do finally roll outside of a repeating pattern that delays results, you get a large number of results all at once.

When converting from sides to sides and back, I measured average delays between and flips. The effect of the delay on our percentage of efficiency limits to as the number of rolls increases.

This is as far as we can go, I think. We never lose any rolls, so entropy is never permanently lost, but there's a small delay. Oh, and the performance is awful. The rational numbers used in the computation don't stay small (unless one of the dice has a number of sides that divides the other die's number of sides). They get bigger and bigger and BIGGER as you generate more rolls. It doesn't take long to convert a thousand rolls, but ten thousand is going to take a minute. Literally.

Summary

You can convert an -sided die to an -sided die without losing any entropy, with the caveat that you expect a constant small delay between having generated enough entropy and yielding the next roll.

This solution is interesting theoretically, but note that in practice you'd be far better off doing naive discarding (the constants work well for coins-to-dice). Entropy is not particularly expensive. You can afford to waste 4% of it.

---

Discuss on Reddit, Hacker News

---