All callers of `byte_size` expect `size_t`. Change its implementation to use pointer_delta, which has built-in checks for to >= from.