Problem description
SHSTK is an x64 security feature that is part of Intel CET. In order to support SHSTK, each object of a library must have a .note.gnu.property section. The problem is, libvmaf's assembly files do not emit this section. As a result, libvmaf.so is not marked as being SHSTK-compatible, and this causes SHSTK not to be supported on applications that use libvmaf.
How to reproduce
The library is not marked as being SHSTK-compatible:
user@fedora:~$ readelf -n /usr/lib64/libvmaf.so.3 | grep SHSTK
... (no results) ...
Proposed fix
Libvmaf builds a single x64 assembly file, libvmaf/src/x86/cpuid.asm, which includes libvmaf/src/ext/x86/x86inc.asm for generic ASM definitions. cpuid.asm does not have mismatched calls/rets, and therefore can support SHSTK as-is, it only needs to emit the .note.gnu.property section to advertise that it is SHSTK-compatible. x86inc.asm comes from VideoLan, which in fact added support for emitting that section in this commit from 2024.
Therefore, fixing the issue is only a matter of syncing x86inc.asm with the upstream VideoLan version. Or in a reduced form, cherry-picking the aforementioned commit. Having done the latter locally, I can confirm that it solves the problem:
user@fedora:~$ readelf -n libvmaf.so.3 | grep SHSTK
Properties: x86 feature: SHSTK
Problem description
SHSTK is an x64 security feature that is part of Intel CET. In order to support SHSTK, each object of a library must have a
.note.gnu.propertysection. The problem is, libvmaf's assembly files do not emit this section. As a result,libvmaf.sois not marked as being SHSTK-compatible, and this causes SHSTK not to be supported on applications that use libvmaf.How to reproduce
The library is not marked as being SHSTK-compatible:
Proposed fix
Libvmaf builds a single x64 assembly file,
libvmaf/src/x86/cpuid.asm, which includeslibvmaf/src/ext/x86/x86inc.asmfor generic ASM definitions.cpuid.asmdoes not have mismatched calls/rets, and therefore can support SHSTK as-is, it only needs to emit the.note.gnu.propertysection to advertise that it is SHSTK-compatible.x86inc.asmcomes from VideoLan, which in fact added support for emitting that section in this commit from 2024.Therefore, fixing the issue is only a matter of syncing
x86inc.asmwith the upstream VideoLan version. Or in a reduced form, cherry-picking the aforementioned commit. Having done the latter locally, I can confirm that it solves the problem: