---
# try also 'default' to start simple
theme: default
# random image from a curated Unsplash collection by Anthony
# like them? see https://unsplash.com/collections/94734566/slidev
# background: https://source.unsplash.com/collection/94734566/1920x1080
# apply any windi css classes to the current slide
class: 'text-center'
# https://sli.dev/custom/highlighters.html
highlighter: shiki
# show line numbers in code blocks
lineNumbers: false
# persist drawings in exports and build
drawings:
  persist: false
# use UnoCSS (experimental)
css: unocss
---

# Compressed debug sections

<!--
The last comment block of each slide will be treated as slide notes. It will be visible and editable in Presenter Mode along with the slide. [Read more in the docs](https://sli.dev/guide/syntax.html#notes)
-->

<style>
h1 {
  background-color: #2B90B6;
  background-image: linear-gradient(45deg, #4EC5D4 10%, #146b8c 20%);
  background-size: 100%;
  -webkit-background-clip: text;
  -moz-background-clip: text;
  -webkit-text-fill-color: transparent;
  -moz-text-fill-color: transparent;
}
</style>

---
layout: 'intro'
---

<h1 text="!5xl">MaskRay</h1>

<div class="leading-8 opacity-80">
<a href="https://maskray.me/portfolio/llvm/">LLVM contributor since 2017</a>, ld.lld and Clang Driver maintainer<br>
binutils, glibc, GCC<br>
</div>

<div class="my-10 grid grid-cols-[40px_1fr] w-min gap-y-4">
  <ri-github-line class="opacity-50"/>
  <div><a href="https://github.com/MaskRay" target="_blank">MaskRay</a></div>
  <ri-user-3-line class="opacity-50"/>
  <div><a href="https://maskray.me" target="_blank">maskray.me</a></div>
</div>

<!-- <img src="/img/me.jpg" class="rounded-full size-200px object-cover-top abs-tr mt-16 mr-12"/> -->

---

# Compressed debug sections

- Binary size is important
- Compressing code and data is usually not suitable
- Filesystem level compression is not sufficiently portable. It does not leverage application information well
- Debug sections are large. Compressing them is profitable

---

## Case study

Here is a `-DCMAKE_BUILD_TYPE=Debug` build directory of llvm-project where I just ran `ninja clang`.

```text
% stat -c %s **/*.o | awk '{s+=$1} END{print s}'
1464767464
% readelf -WS **/*.o | awk 'BEGIN{FPAT="\\[.*?\\]|\\S+"} $2~/\.text/{d += strtonum("0x"$6)} END{print d}'
210026370
% readelf -WS **/*.o | awk 'BEGIN{FPAT="\\[.*?\\]|\\S+"} $2~/\.debug_/{d += strtonum("0x"$6)} END{print d}'
631069751
% readelf -WS **/*.o | awk 'BEGIN{FPAT="\\[.*?\\]|\\S+"} $2~/\.rela\.debug_/{d += strtonum("0x"$6)} END{print d}'
78448968
```

It is typical that the debug information is often much larger than text sections.

---

## `.zdebug` format

In 2007-11, Craig Silverstein [added `--compress-debug-sections=zlib` to gold](https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=9a0910c33e1a6962d475ee0a994fd1f5e446a888).
When the option was specified, gold compressed the content of a `.debug*` section with zlib and changed the section name to `.debug*.zlib.$uncompressed_size`.

In 2008-04, Craig Silverstein [changed the format](https://sourceware.org/pipermail/binutils/2008-April/055837.html) and contributed [Patch to handle compressed sections](https://sourceware.org/pipermail/gdb-patches/2008-March/056449.html) to gdb.
The compressed section was renamed to `.zdebug*`.

In 2010-06, Cary Coutant [added `--compress-debug-sections` to gas](https://sourceware.org/pipermail/binutils/2010-June/067606.html) and added reading support to objdump and readelf.

---

## Generic ABI `SHF_COMPRESS`

In 2012, Solaris 11 introduced ancillary objects and found needs to compress debug sections.
They studied the `.zdebug` format and identified some format problems (magic section prefix, big-endian integer is not ELF like, unable to describe differing alignment requirements).

Ali Bahrami proposed `SHF_COMPRESS` to the generic ABI and got it standardized.
See [ELF Section Compression](http://www.linker-aliens.org/blogs/ali/entry/elf_section_compression/) for a nice summary

---

## Usage

To use compressed debug sections, just remember one GCC/Clang driver option `-gz`(or a variant like `-gz=zlib`). The option combines two tasks.

---

<style>
.slidev-layout code {
  line-height: 14px;
}
</style>

## Assembling

```text
% clang -c -g -gz a.c
% clang -c -g -gz=zlib a.c
```

For object generation, it acts as `-Wa,--compress-debug-sections=zlib` and asks the assembler to compress debug sections in the output `.o` (and `.dwo` when `-gsplit-dwarf` is specified).
A compressed section has the `SHF_COMPRESSED` flag and its content begins with a compression header structure that identifies the compression algorithm.
```c
typedef struct {
	Elf32_Word	ch_type;
	Elf32_Word	ch_size;
	Elf32_Word	ch_addralign;
} Elf32_Chdr;

typedef struct {
	Elf64_Word	ch_type;
	Elf64_Word	ch_reserved;
	Elf64_Xword	ch_size;
	Elf64_Xword	ch_addralign;
} Elf64_Chdr;
```

As of 2022, only `ELFCOMPRESS_ZLIB` and `ELFCOMPRESS_ZSTD` are defined:

> ELFCOMPRESS_ZLIB - The section data is compressed with the ZLIB algorithm. The compressed ZLIB data bytes begin with the byte immediately following the compression header, and extend to the end of the section. Additional documentation for ZLIB may be found at http://zlib.net.
>
> ELFCOMPRESS_ZSTD - The section data is compressed with the Zstandard algoritm. The compressed Zstandard data bytes begin with the byte immediately following the compression header, and extend to the end of the section. Additional documentation for Zstandard may be found at http://www.zstandard.org

Note that `-gz` does not imply emit debug information, which is controlled by `-g`. So you generally need both `-g -gz`.
This feature orthogonality is convenient when a subset of object files need to opt out debug information emission.

---

## Linking

```text
% clang -gz a.o
% clang -gz=zlib a.o
```

For linking, `-gz` acts as `-Wl,--compress-debug-sections=zlib` and asks the linker to compress debug sections in the linked image.
If you need decompression for linker input but don't need compression for the linker output, don't bother with `-gz`.
The linker recognizes compressed input and decompresses it automatically.

You may not want to use `-gz` if you combine assembly and linking in one step (`gcc -g -gz a.c` without `-S` or `-c`).
The intermediate `.o` file will be discarded. The assembler compressed debug sections will immediately be decompressed by the linker, causing wasted efforts.

binutils and GCC support `zlib-gabi`. This was added to parallel the renamed legacy format `zlib-gnu`.
Nowadays just use `zlib` and avoid `zlib-gabi`. ELF compresion has been standardized and new formats (e.g. zstd) do not use `-gabi`.

---

## zlib => Zstandard

* zlib-madler is slow.
* Some forks (e.g. zlib-cloudflare and zlib-ng) are faster (the output may be slightly larger) but still not as good as zstd.
* zstd is better than zlib in all metrics: compression speed, decompression speed, and compression ratio.

The ecosystem issue required significant undertaking and stakeholder buy-in.

* [[RFC] Zstandard as a second compression method to LLVM](https://discourse.llvm.org/t/rfc-zstandard-as-a-second-compression-method-to-llvm/63399) in June 2022
* [Add new ch_type value: ELFCOMPRESS_ZSTD](https://groups.google.com/g/generic-abi/c/satyPkuMisk)

---

Personal opinion on required properties

* It has an open compression algorithm and implementation.
* It provides significant benefits (compression speed, decompression speed, compression ratio) with a decent memory footprint and complexity.
* It has full backward compatibility. In 20 years I want to be able to decompress a debug section created today.
* It has a wide range of and active use cases. When the format value is standardized, consumers are willing to add support.
* It has good documentation.
* It's easy to use.

---

<style>
.slidev-layout code {
  line-height: 10px;
}
</style>

```text
% numactl -C 20 ./bench.sh debug_info
comp    RSS     decomp  RSS     size    command
19.99   2568    2.37    2212    212342615       pigz -zfk -p 1 -S .o0 debug_info
2.44    4044    2.48    19188   244890939       brotli -f -q 1 debug_info -o debug_info.o0
5.92    36300   2.22    19264   211420890       brotli -f -q 3 debug_info -o debug_info.o0
22.77   110516  2.11    21128   193129564       brotli -f -q 5 debug_info -o debug_info.o0
239.99  114140  2.12    20988   191699881       brotli -f -q 9 debug_info -o debug_info.o0
36.24   2396    15.71   1604    212700609       bzip2 -cf -1 debug_info
34.29   3640    16.52   2396    221881700       bzip2 -cf -3 debug_info
9.16    2008    0.14    2604    227302010       gzip -fk -S .o0 -1 debug_info
11.57   2072    0.14    2536    221444175       gzip -fk -S .o0 -3 debug_info
1.16    8872    0.49    9116    310918343       lz4 -fq --fast debug_info debug_info.o0
1.25    8828    0.48    9168    295430369       lz4 -fq -1 debug_info debug_info.o0
6.85    8904    0.49    8896    265793678       lz4 -fq -3 debug_info debug_info.o0
8.60    9016    0.49    9000    263660439       lz4 -fq -5 debug_info debug_info.o0
14.26   8936    0.48    8848    262365618       lz4 -fq -9 debug_info debug_info.o0
1.42    2244    1.24    1720    283798959       lzop -f -1 debug_info -o debug_info.o0
1.43    2088    1.25    1632    282736380       lzop -f -3 debug_info -o debug_info.o0
71.94   2504    1.32    1756    240402339       lzop -f -9 debug_info -o debug_info.o0
27.76   5024    11.23   2560    164560076       xz -c -0 debug_info
34.19   10992   10.97   3292    163894892       xz -c -1 debug_info
56.63   34032   10.83   6296    164369948       xz -c -3 debug_info
149.35  97648   10.75   10400   149764760       xz -c -6 debug_info
2.55    12696   0.70    2916    252027101       zstd -fq --fast debug_info -o debug_info.o0
2.97    12528   0.83    2964    234661626       zstd -fq -1 debug_info -o debug_info.o0
3.95    41764   0.90    4568    215717860       zstd -fq -3 debug_info -o debug_info.o0
5.93    43960   0.91    4572    213222742       zstd -fq -5 debug_info -o debug_info.o0
9.96    90436   0.89    6668    208886078       zstd -fq -9 debug_info -o debug_info.o0
110.59  198708  1.46    10588   178996922       zstd -fq -18 debug_info -o debug_info.o0
```

---

## Toolchain support

The next step is to add toolchain support. The most important pieces are assemblers, linkers, and debuggers.
Many other pieces are needed as well.

* binutils-gdb, gcc
* llvm-project
* elfutils
* mold
* dwz, bloaty, etc
* go, rust, etc

---

## llvm-project support

* llvm-project: all implemented as of 2022-09 (milestone: 16.0.0).
  + Clang: compress `.o` and (if split DWARF is enabled) `.dwo` with level 5
  + llvm-objcopy: `--decompress-debug-sections` and `--compress-debug-sections=zstd` (level 5). Implemented in [D130458](https://reviews.llvm.org/D130458) (ELFCLASS64) and [D134385](https://reviews.llvm.org/D134385) (ELFCLASS32)
  + ld.lld: decompress `ELFCOMPRESS_ZSTD` input sections ([D129406](https://reviews.llvm.org/D129406)) and compress output debug sections with level 3 ([D133548](https://reviews.llvm.org/D133548), [D133679](https://reviews.llvm.org/D133679))
  + llvm-dwarfdump, llvm-dwp, llvm-symbolizer, lldb: use LLVMObject API to decompress `ELFCOMPRESS_ZSTD` input sections ([D134116](https://reviews.llvm.org/D134116))

Steps

* CMake change for `LLVM_ENABLE_ZSTD`, `llvm/Support/Compression.cpp`
* llvm-objcopy: `--decompress-debug-sections` and `--compress-debug-sections=zstd`
* LLVMMC (integrated assembler), LLVMObject (llvm-dwarfdump, llvm-dwp, llvm-symbolizer, lldb), lld
* Clang `-gz=zstd`

---

<style>
.slidev-layout code {
  line-height: 12px;
}
</style>

## GNU toolchain support

* binutils
  + addr2line: symbolization needs to decompress debug sections
  + gas: compress debug sections
  + ld, gold: decompress compressed input sections and compress output debug sections. [gold feature request](https://sourceware.org/bugzilla/show_bug.cgi?id=29641)
  + dwp: decompress compressed `.dwo`. dwp uses gold's code
  + nm: `--line-numbers` uses debug information
  + objcopy: `--decompress-debug-sections` and `--compress-debug-sections=zstd`
  + objdump: `--dwarf` decompresses compressed debug sections
  + readelf: `--debug-dump` and `--decompress` decompress compressed sections. [feature request](https://sourceware.org/bugzilla/show_bug.cgi?id=29640)
* gdb
  + decompress compressed debug sections in executables, shared objects, separate debug files, and `.dwo` files. [Feature request](https://sourceware.org/bugzilla/show_bug.cgi?id=29563)
  + MiniDebugInfo section `.gnu_debugdata` is compressed with xz. [zstd feature request](https://sourceware.org/bugzilla/show_bug.cgi?id=29584)
* GCC: [`-gz=zstd` feature request](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106897)

The main changes were for the binutils-gdb repository. This work turned out to be much more challenging than my work for llvm-project.

---

<style>
.slidev-layout code {
  line-height: 12px;
}
</style>

The entry points of zstd compression features were in binutils, gas, and ld. binutils and ld use bfd, so we needed to update bfd.

Created `config/zstd.m4` by following `config/zlib.m4`. Add `AC_ZSTD` and `ZLIB_LIBS` to every top-level project which uses bfd.

```text
% rg -l --sort=path ZSTD_LIBS
bfd/Makefile.am
bfd/Makefile.in
bfd/configure
binutils/Makefile.am
binutils/Makefile.in
binutils/configure
gas/Makefile.am
gas/Makefile.in
gas/configure
gdb/Makefile.in
gdb/acinclude.m4
gdb/configure
ld/Makefile.am
ld/Makefile.in
ld/configure
libctf/Makefile.in
libctf/configure
libctf/configure.ac
sim/Makefile.in
sim/arch-subdir.mk.in
sim/common/Make-common.in
sim/configure
sim/ppc/Makefile.in
```

Remember to update auto-generated files with the appropriate versions of autoconf and automake:
```sh
PATH=~/projects/automake-1.15.1/bin:$PATH ~/projects/autoconf-2.69/bin/autoreconf -vf bfd binutils gas ld libctf sim
```