[BUG]: py::smart_holder does not preserve shared_ptr ownership across casts (weak_ptr expires unexpectedly)

[sergeyy32]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

3.0.2

Problem description

Hello,

When using py::smart_holder as the holder type, casting a Python-owned object to std::shared_ptr does not seem to preserve shared ownership across casts.

As a result, a std::weak_ptr stored from a previous call cannot be locked later, even though the Python object is still alive.

With the classic std::shared_ptr holder, the behavior works as expected.

Below is a minimal reproducible example.

#include <pybind11/pybind11.h>
#include <iostream>
#include <memory>

namespace py = pybind11;

class Base : public std::enable_shared_from_this<Base> {
public:
    Base() = default;
    virtual ~Base() = default;
};

template<class PyExtBase = Base>
class PyBase
    : public PyExtBase
    , public py::trampoline_self_life_support {
public:
    using PyExtBase::PyExtBase;
};

PYBIND11_MODULE(example, m) {
    py::classh<Base, PyBase<>>(m, "Base")
        .def(py::init<>());

    m.def("get_base", [](py::type type) {
        auto b = py::cast<std::shared_ptr<Base>>(type());
        return b;
    });

    m.def("hold_weak", [](std::shared_ptr<Base> b) {
        static std::weak_ptr<Base> b_weak;

        if (auto locked = b_weak.lock()) {
            std::cout << "Current weak pointer use count: "
                      << locked.use_count() << std::endl;
        } else {
            std::cout << "No weak pointer provided." << std::endl;
        }

        b_weak = b;
    });
}

Python usage

import example

class DerivedClass(example.Base):
    def __init__(self):
        super().__init__()

b = example.get_base(DerivedClass)
example.hold_weak(b)
example.hold_weak(b)

Observed output (with py::smart_holder):

No weak pointer provided.
No weak pointer provided.

Observed output (with classic holder)

py::class_<Base, std::shared_ptr<Base>, PyBase<>>(m, "Base")

The output becomes:

No weak pointer provided.
Current weak pointer use count: 4

Is this the intended behavior of py::smart_holder, or could this be a bug?

If this is expected behavior, what would be the recommended way to preserve std::weak_ptr semantics when ownership is managed via smart_holder?

Reproducible example code

Is this a regression? Put the last known working version here if it is.

Not a regression

[sergeyy32]

I also came across this commit:
1b9f85a

From a quick look, it seems that the current behavior might be intentional, possibly as part of the smart_holder design.

However, I’m not fully confident in my understanding of the underlying design decisions here, so I may be missing some important details.