gh-101178: Add Ascii85, base85, and Z85 support to binascii

[kangtastic]

Synopsis

Add Ascii85 and base85 encoder and decoder functions implemented in C to binascii and use them to greatly improve the performance and reduce the memory usage of the existing Ascii85, base85, and Z85 codec functions in base64.

No API or documentation changes are necessary with respect to any functions in base64, and all existing unit tests for those functions continue to pass without modification.

Resolves: gh-101178

Discussion

The base85-related functions in base64 are now wrappers for the new functions in binascii, as envisioned in the docs:

The binascii module contains a number of methods to convert between binary and various ASCII-encoded binary representations. Normally, you will not use these functions directly but use wrapper modules like uu or base64 instead. The binascii module contains low-level functions written in C for greater speed that are used by the higher-level modules.

Parting out Ascii85 from base85 and Z85 was warranted in my testing despite the code duplication due to the various performance-murdering special cases in Ascii85.

Comments and questions are welcome.

Benchmarks

Updated April 20, 2025.

# bench_b85.py

# Note: EXTREMELY SLOW on unmodified mainline CPython
#       when tracing malloc on the base-85 functions.

import base64
import sys
import timeit
import tracemalloc

funcs = [(base64.b64encode, base64.b64decode),  # sanity check/comparison
         (base64.a85encode, base64.a85decode),
         (base64.b85encode, base64.b85decode),
         (base64.z85encode, base64.z85decode)]

def mb(n):
    return f"{n / 1024 / 1024:.3f} MB"

def stats(func, data, t, m):
    name, n, bps = func.__qualname__, len(data), len(data) / t
    print(f"{name} : {n} b in {t:.3f} s ({mb(bps)}/s) using {mb(m)}")

if __name__ == "__main__":
    data = b"a" * int(sys.argv[1]) * 1024 * 1024
    for fenc, fdec in funcs:
        tracemalloc.start()
        enc = fenc(data)
        menc = tracemalloc.get_traced_memory()[1] - len(enc)
        tracemalloc.stop()
        tenc = timeit.timeit("fenc(data)", number=1, globals=globals())
        stats(fenc, data, tenc, menc)

        tracemalloc.start()
        dec = fenc(enc)
        mdec = tracemalloc.get_traced_memory()[1] - len(dec)
        tracemalloc.stop()
        tdec = timeit.timeit("fdec(enc)", number=1, globals=globals())
        stats(fdec, enc, tdec, mdec)

# Python 3.14.0a7+ commit 78cfee6f09
# ./configure --enable-optimizations --with-lto

# With this PR
$ time ./python bench_b85.py 64
b64encode : 67108864 b in 0.084 s (763.340 MB/s) using 42.667 MB
b64decode : 89478488 b in 0.230 s (371.074 MB/s) using 56.889 MB
a85encode : 67108864 b in 0.190 s (336.115 MB/s) using 0.000 MB
a85decode : 83886080 b in 0.216 s (370.605 MB/s) using 0.000 MB
b85encode : 67108864 b in 0.072 s (887.955 MB/s) using 0.000 MB
b85decode : 83886080 b in 0.175 s (457.224 MB/s) using 0.000 MB
z85encode : 67108864 b in 0.072 s (891.721 MB/s) using 0.000 MB
z85decode : 83886080 b in 0.174 s (460.582 MB/s) using 0.000 MB

real    0m2.231s
user    0m2.064s
sys     0m0.156s

# Unmodified
$ time ./python bench_b85.py 64
b64encode : 67108864 b in 0.082 s (781.718 MB/s) using 42.667 MB
b64decode : 89478488 b in 0.237 s (360.686 MB/s) using 56.889 MB
a85encode : 67108864 b in 7.492 s (8.543 MB/s) using 2664.406 MB
a85decode : 83886080 b in 14.264 s (5.609 MB/s) using 3332.254 MB
b85encode : 67108864 b in 7.181 s (8.912 MB/s) using 2664.404 MB
b85decode : 83886080 b in 8.486 s (9.427 MB/s) using 3332.254 MB
z85encode : 67108864 b in 7.343 s (8.715 MB/s) using 2664.102 MB
z85decode : 83886080 b in 8.778 s (9.113 MB/s) using 3332.254 MB

real    9m2.346s
user    8m47.248s
sys     0m12.460s

The old pure-Python implementation is two orders of magnitude slower and uses over O(40n) temporary memory.

[ghost]

All commit authors signed the Contributor License Agreement.

[kangtastic]

It's a year later, and Z85 support has been added to base64 in the meantime. So while bringing this PR up to date with main, I added Z85 support to it as well.

For reference, this is the benchmark run that led me to do so.

# After merging main but before adding Z85 support to this PR
(cpython-b85) $ python bench_b85.py 64
b64encode : 67108864 b in 0.121 s (527.435 MB/s) using 42.667 MB
b64decode : 89478488 b in 0.309 s (276.188 MB/s) using 56.889 MB
a85encode : 67108864 b in 0.297 s (215.150 MB/s) using 0.000 MB
a85decode : 83886080 b in 0.205 s (390.751 MB/s) using 0.000 MB
b85encode : 67108864 b in 0.106 s (604.359 MB/s) using 0.000 MB
b85decode : 83886080 b in 0.204 s (393.040 MB/s) using 0.000 MB
z85encode : 67108864 b in 0.204 s (313.610 MB/s) using 80.000 MB
z85decode : 83886080 b in 0.300 s (266.670 MB/s) using 100.000 MB

The existing Z85 implementation translates from the standard base85 alphabet to Z85 after the fact and within Python, so it was already benefiting from this PR but with substantial performance and memory usage overhead. That overhead is now gone.

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[kangtastic]

PR has been rebased onto main at 78cfee6 with squashing.

[sergey-miryanov]

Note that attempting to decode Ascii85, base85, or Z85 data of length 1 mod 5 now produces an error, as no encoder would emit such data. This should be the only significant externally visible difference compared to the old implementations.

I believe you have to document this change.

[kangtastic]

Note that attempting to decode Ascii85, base85, or Z85 data of length 1 mod 5 now produces an error, as no encoder would emit such data. This should be the only significant externally visible difference compared to the old implementations.

I believe you have to document this change.

Fair point, I could do that.

In case anyone argues for keeping the old behavior (silently ignoring length 1 mod 5), I won't do it just yet.

[kangtastic]

The PR has been updated to preserve the existing base 85 Python functions in base64 and modify the new base 85 C functions in binascii to closely match their behavior. Notably, trying to decode data of length 1 mod 5 is no longer an error.

[kangtastic]

PR was accidentally closed due to misclicking on mobile. There should be a confirmation dialog or something 😅