Skip to content

DLPX-96700 Re-enable delphix/bcc package to linux-pkg#389

Open
tonynguien wants to merge 1 commit intodevelopfrom
dlpx/pr/tonynguien/95f8f397-6d1f-4dc3-aa29-b8f1bd5545b1
Open

DLPX-96700 Re-enable delphix/bcc package to linux-pkg#389
tonynguien wants to merge 1 commit intodevelopfrom
dlpx/pr/tonynguien/95f8f397-6d1f-4dc3-aa29-b8f1bd5545b1

Conversation

@tonynguien
Copy link
Contributor

@tonynguien tonynguien commented Mar 2, 2026

Background

This is a second attempt of #383 and I've tested with the updated delphix/bcc repo, see delphix/bcc#20

Problem

estat commands with zfs headers are failing, i.e. estat zio | zil | zpl | zvol | nfs-by-client | metaslab-alloc, see https://perforce.atlassian.net/browse/DLPX-96293.

The problem is Ubuntu bpfcc-tools and related packages the two years old, version 0.29, and support up to the 6.8 kernel (the main Ubuntu 24.04). Our images are running 6.14 kernel thus we get below forward declaration error

In file included from include/linux/security.h:35:
include/linux/bpf.h:352:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_wq'
  352 |                 return sizeof(struct bpf_wq);
      |                        ^     ~~~~~~~~~~~~~~~
include/linux/bpf.h:352:24: note: forward declaration of 'struct bpf_wq'
  352 |                 return sizeof(struct bpf_wq);
      |                                      ^
include/linux/bpf.h:382:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_wq'
  382 |                 return __alignof__(struct bpf_wq);
      |                        ^          ~~~~~~~~~~~~~~~
include/linux/bpf.h:382:29: note: forward declaration of 'struct bpf_wq'
  382 |                 return __alignof__(struct bpf_wq);
      |                                           ^
4 warnings and 2 errors generated.
Traceback (most recent call last):
  File "/usr/share/performance-diagnostics/bpf/standalone/zil.py", line 316, in <module>
    b = BPF(text=bpf_text,
        ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 479, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))

I’ve verified that a recent build of iovisor/bcc works cleanly and we no longer get these warning/error.

Solution

The solution is to build our own bcc packages which is what we did prior to the upgrade to 24.04 LTS. The delphix/bcc is a couple years out of date so the related delphix/bcc#20 PR updates that repo and adds minor changes to properly build and rename Debian packages so they can cleanly replace Ubuntu bpfcc packages.

Testing Done

Successful build against the delphix/bcc with merge changes.

delphix@ip-10-110-243-71:~ export DEFAULT_GIT_BRANCH=dlpx/pr/tonynguien/5047efc9-172e-480b-80ce-86a59c7935e7
delphix@ip-10-110-243-71:~ export DELPHIX_RELEASE_VERSION=2026.1.0.0
delphix@ip-10-110-243-71:~ ./buildpkg.sh bcc 2>&1 | tee build.delphix
delphix@ip-10-110-243-71:~/linux-pkg$ tail -n20 build.delphix

Running: cd /export/home/delphix/linux-pkg/packages/bcc/tmp

PACKAGE bcc: STAGE post_build_checks STARTED
Running: post_build_checks
Running: dpkg-deb -c bpfcc-lua_0.36.1-1+delphix.2026.02.27.22.53_all.deb | grep '/usr/share/doc/' | grep copyright
-rw-r--r-- root/root       193 2026-02-27 22:53 ./usr/share/doc/bpfcc-lua/copyright
Running: dpkg-deb -c bpfcc-tools_0.36.1-1+delphix.2026.02.27.22.53_all.deb | grep '/usr/share/doc/' | grep copyright
-rw-r--r-- root/root       193 2026-02-27 22:53 ./usr/share/doc/bpfcc-tools/copyright
Running: dpkg-deb -c libbpfcc-examples_0.36.1-1+delphix.2026.02.27.22.53_amd64.deb | grep '/usr/share/doc/' | grep copyright
-rw-r--r-- root/root       193 2026-02-27 22:53 ./usr/share/doc/libbpfcc-examples/copyright
Running: dpkg-deb -c libbpfcc_0.36.1-1+delphix.2026.02.27.22.53_amd64.deb | grep '/usr/share/doc/' | grep copyright
-rw-r--r-- root/root       193 2026-02-27 22:53 ./usr/share/doc/libbpfcc/copyright
Running: dpkg-deb -c python3-bpfcc_0.36.1-1+delphix.2026.02.27.22.53_all.deb | grep '/usr/share/doc/' | grep copyright
-rw-r--r-- root/root       193 2026-02-27 22:53 ./usr/share/doc/python3-bpfcc/copyright
PACKAGE bcc: STAGE post_build_checks COMPLETED in 1 seconds

Success: Package bcc has been built successfully.
Build products are in /export/home/delphix/linux-pkg/packages/bcc/tmp/artifacts

Testing BCC programs work

Sampling at 49 Hertz of all threads by user + kernel stack for 1 secs.
^C
    __raw_spin_unlock_irq+0x10 [kernel]
    __raw_spin_unlock_irq+0x10 [kernel]
    process_one_work+0x15d [kernel]
    worker_thread+0x2e5 [kernel]
    kthread+0xe7 [kernel]
    ret_from_fork+0x44 [kernel]

Some estat commands are still failing though that will be separately tracked/addressed with https://perforce.atlassian.net/browse/DLPX-96701

delphix@ip-10-110-223-173:~$ sudo estat zpl 2
In file included from /virtual/main.c:62:
In file included from /usr/src/zfs-6.14.0-1018-dx2025121319-a7d298781-aws/include/sys/spa_impl.h:42:
/usr/src/zfs-6.14.0-1018-dx2025121319-a7d298781-aws/include/sys/vdev_raidz.h:43:8: error: redefinition of 'kernel_param'
   43 | struct kernel_param {};
      |        ^
include/linux/moduleparam.h:69:8: note: previous definition is here
   69 | struct kernel_param {
      |        ^
/virtual/main.c:102:2: error: use of undeclared identifier 'zfsvfs_t'
  102 |         zfsvfs_t *zfsvfs = zn->z_inode.i_sb->s_fs_info;
      |         ^
/virtual/main.c:102:12: error: use of undeclared identifier 'zfsvfs'
  102 |         zfsvfs_t *zfsvfs = zn->z_inode.i_sb->s_fs_info;
      |                   ^
/virtual/main.c:102:23: error: incomplete definition of type 'struct znode'
  102 |         zfsvfs_t *zfsvfs = zn->z_inode.i_sb->s_fs_info;
      |                            ~~^
/usr/src/zfs-6.14.0-1018-dx2025121319-a7d298781-aws/include/sys/zfs_acl.h:205:8: note: forward declaration of 'struct znode'
  205 | struct znode;
      |        ^
/virtual/main.c:104:19: error: use of undeclared identifier 'zfsvfs'
  104 |         objset_t *z_os = zfsvfs->z_os;
      |                          ^
5 errors generated.
Traceback (most recent call last):
  File "/usr/bin/estat", line 414, in <module>
    b = BPF(text=bpf_text, cflags=cflags, debug=debug_level)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 507, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

@tonynguien tonynguien force-pushed the dlpx/pr/tonynguien/95f8f397-6d1f-4dc3-aa29-b8f1bd5545b1 branch from e8e73fa to b89ddc0 Compare March 2, 2026 19:58
@tonynguien tonynguien marked this pull request as ready for review March 2, 2026 21:09
Copy link

@dbjwhs-perforce dbjwhs-perforce left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

#
# shellcheck disable=SC2034

#

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit question: empty comment block?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants