cf_pcap fails with "failed to open device lo"

[metskem]

Current behavior

We are trying to use the new cf-pcap feature, but issuing the command gives the following response:

`/tmp/lifecycle/cf-pcap -interface lo 'port 8080'

time=2026-03-11T15:15:43.589+01:00 level=ERROR msg="failed to set up capture" error="failed to open device lo: lo: You don't have permission to perform this capture on that device (Attempt to create packet socket failed - CAP_NET_RAW may be required)"`

We are running diego-release 2.130.0 on Ubuntu Noble.

We have set the enable_cf_pcap flag to true:

bosh int /tmp/cf-manifest.yml --path /instance_groups/name=diego-cell/jobs/name=rep/properties/rep enable_cf_pcap: true

Desired behavior

It should start the packet capture.

Affected Version

2.130.0

[metskem]

Tagging @maxmoehl

[maxmoehl]

I think I know what is going on. There is a cache on the diego-cell which contains assets such as the cf-pcap binary. Only when the flag is set will the extraction of the downloaded archive preserve the additional file attributes required to enable the user to run captures. Apparently in my test case there was also a change in the stem cell causing the VM to be recreated and the cache to be destroyed. After starting again all files were downloaded and extracted cleanly. In your case you probably only restarted the jobs without recreating the VM meaning the previous cache was used as-is.

I will have to see how I can adjust the caching to take into account such side-effects.

[maxmoehl]

I have a PR with a potential fix, still have to validate it on one of our envs.

[maxmoehl]

Okay, the env I have available to me right now is on 2.126.0, so I had to back-port the changes to that version and create a dev release, this shouldn't have any effect as the feature is pretty stand-alone. So my steps were:

1. Deploy the dev-release without setting the pcap flag.
2. Check that the binary exists and cannot capture.
3. Switch only the flag to true and deploy again.
4. Check that the binary exists and can capture.
5. Switch only the flag to false and deploy again.
6. Check that the binary exists and cannot capture.

I can see that the file which tracks the flag is created at step one and it properly wipes the cache (by deleting the cache state JSON) when the flag is flipped at step 3. I could then capture packets using cf ssh $app -c ... and view them in WireShark. Setting the flag back to false caused the capture to fail again.

Note that I didn't have to do anything on CF side, the deploy of the cells is sufficient to inject the correct binaries into the container.

I've marked the PR as ready for review.

[metskem]

In the meantime we had a cf environment repaved with a newer stemcell, I noticed the problem is still there.
Is there anything on my side that I can check?
We are running Ubuntu Noble, could that be related?

[maxmoehl]

Hm that's interesting. Can you provide the following information from one of your diego-cells:

# cat /var/vcap/jobs/rep/config/bpm.yml
# getpcaps $(pidof rep)
# getcap /var/vcap/data/rep/shared/garden/download_cache/*/cf-pcap

In the rep bpm config you should see the capability SETFCAP, that one should also show for the getpcaps command. The last command should show CAP_NET_RAW for cf-pcap.

[maxmoehl]

Ohh, I just noticed that /var/vcap/data is a separate disk. Now I'm no longer sure how I managed to getting it to work. Maybe I somehow wiped that as well, I did quite a lot of fiddling around to get it to work initially 😅

[metskem]

It indeed doesn't have the SETFCAP capability:

diego-cell/d94ca478-435d-4cae-9f4d-ea19ffcceb16:/var/vcap/jobs/rep/config# cat /var/vcap/jobs/rep/config/bpm.yml
---
processes:
- name: rep
  executable: /var/vcap/jobs/rep/bin/rep
  limits:
    open_files: 100000
  hooks:
    pre_start: /var/vcap/jobs/rep/bin/bpm-pre-start
  ephemeral_disk: true
  additional_volumes:
    - path: /var/vcap/data/voldrivers
    - path : /var/vcap/data/garden
      writable: true
    - path: /var/vcap/store/rootfses
diego-cell/d94ca478-435d-4cae-9f4d-ea19ffcceb16:~# getpcaps $(pidof rep)
13413: =
diego-cell/d94ca478-435d-4cae-9f4d-ea19ffcceb16:~# getcap /var/vcap/data/rep/shared/garden/download_cache/*/cf-pcap
diego-cell/d94ca478-435d-4cae-9f4d-ea19ffcceb16:~#

[maxmoehl]

Looking at your initial comment more closely:

bosh int /tmp/cf-manifest.yml --path /instance_groups/name=diego-cell/jobs/name=rep/properties/rep enable_cf_pcap: true

The exact name of the property is diego.rep.enable_cf_pcap, it's nested below diego.rep instead of being a top-level rep property. This then triggers the flip in BPM, as long as that's not there the issue is somewhere with the property setting.

[metskem]

Dang, me bad, sorry for all the fuzz...
I corrected the property, bosh deployed and recreated a diego-cell, and then it works like charm.

Thanks for the support!