Skip to content

Fix memory leak if ossl_bn_new() fails#1010

Open
ndossche wants to merge 1 commit intoruby:masterfrom
ndossche:clesss-2
Open

Fix memory leak if ossl_bn_new() fails#1010
ndossche wants to merge 1 commit intoruby:masterfrom
ndossche:clesss-2

Conversation

@ndossche
Copy link

@ndossche ndossche commented Feb 24, 2026

When that call fails, the bn BIGNUM is never freed in asn1integer_to_num(). To solve this, use rb_protect().

Example Valgrind report:

32 (24 direct, 8 indirect) bytes in 1 blocks are definitely lost in loss record 11,113 of 25,910
  malloc (at /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
  CRYPTO_zalloc (at /usr/lib/x86_64-linux-gnu/libcrypto.so.3)
  BN_new (at /usr/lib/x86_64-linux-gnu/libcrypto.so.3)
  BN_bin2bn (at /usr/lib/x86_64-linux-gnu/libcrypto.so.3)
  <unknown stack frame>
 *asn1integer_to_num (ossl_asn1.c:136)
 *asn1integer_to_num_i (ossl_asn1.c:165)
  rb_protect (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
 *decode_int (ossl_asn1.c:356)
 *int_ossl_asn1_decode0_prim (ossl_asn1.c:777)
 *ossl_asn1_decode0 (ossl_asn1.c:936)
 *ossl_asn1_decode_all (ossl_asn1.c:1058)
  <unknown stack frame>
  <unknown stack frame>
  <unknown stack frame>
  rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  rb_catch_obj (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  <unknown stack frame>
  rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_yield (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_ary_each (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  <unknown stack frame>
  rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_yield (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_ary_each (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  <unknown stack frame>
  rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_yield (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_ary_each (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  <unknown stack frame>
  rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  rb_catch_obj (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  <unknown stack frame>
  rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_vm_invoke_proc (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_proc_call_kw (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)

This was found by a hybrid static-dynamic analyser that looks for inconsistent handling of error checks in bindings.

When that call fails, the `bn` BIGNUM is never freed in
asn1integer_to_num(). To solve this, use rb_protect().

Example Valgrind report:
```
32 (24 direct, 8 indirect) bytes in 1 blocks are definitely lost in loss record 11,113 of 25,910
  malloc (at /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
  CRYPTO_zalloc (at /usr/lib/x86_64-linux-gnu/libcrypto.so.3)
  BN_new (at /usr/lib/x86_64-linux-gnu/libcrypto.so.3)
  BN_bin2bn (at /usr/lib/x86_64-linux-gnu/libcrypto.so.3)
  <unknown stack frame>
 *asn1integer_to_num (ossl_asn1.c:136)
 *asn1integer_to_num_i (ossl_asn1.c:165)
  rb_protect (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
 *decode_int (ossl_asn1.c:356)
 *int_ossl_asn1_decode0_prim (ossl_asn1.c:777)
 *ossl_asn1_decode0 (ossl_asn1.c:936)
 *ossl_asn1_decode_all (ossl_asn1.c:1058)
  <unknown stack frame>
  <unknown stack frame>
  <unknown stack frame>
  rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  rb_catch_obj (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  <unknown stack frame>
  rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_yield (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_ary_each (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  <unknown stack frame>
  rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_yield (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_ary_each (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  <unknown stack frame>
  rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_yield (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_ary_each (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  <unknown stack frame>
  rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  rb_catch_obj (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  <unknown stack frame>
  <unknown stack frame>
  <unknown stack frame>
  rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_vm_invoke_proc (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
  rb_proc_call_kw (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant