[Bug]: `SummaryDoc.structure.site_properties` contains `selective_dynamics` (and other keys) filled with None, which breaks pymatgen → ASE conversion

[Sankek]

Code snippet

from mp_api.client import MPRester
from pymatgen.io.ase import AseAtomsAdaptor

with MPRester(MP_API_KEY) as mpr:
    mp_docs = mpr.materials.summary.search()

s = mp_docs[0].structure

print(s.site_properties)  # {'charge': [None, ... None], 'selective_dynamics': [None, ...}
AseAtomsAdaptor.get_atoms(s)  # crashes

What happened?

Description

I’m using mp-api==0.45.15 to fetch SummaryDoc objects from Materials Project. The returned pymatgen.Structure objects contain site_properties keys with lists of None values (e.g. selective_dynamics=[None, ...]), and this crashes ASE conversion via pymatgen.io.ase.AseAtomsAdaptor.

Error:

TypeError: bad operand type for unary ~: 'NoneType'
  File .../pymatgen/io/ase.py:184, in get_atoms
    cmask_site = (~np.array(selective_dynamics)).tolist()

Warnings during fetch

These warnings appear during mpr.materials.summary.search() (even before I try converting to ASE). They suggest that the incoming documents include site properties that are missing for some sites, and pymatgen fills the missing entries with None. That “None-filled” selective_dynamics then propagates into the Structure object and triggers the ASE conversion crash shown above.

/.../mp_api/client/core/client.py:1326: UserWarning: Omitting a query also includes deprecated documents in the results. Make sure to post-filter them out.
  warnings.warn(
/.../pymatgen/core/structure.py:2917: UserWarning: Not all sites have property selective_dynamics. Missing values are set to None.
  return cls.from_sites(...)
(and similar warnings for charge/velocities/coordination_no/forces/magmom)

Expected behavior

Either:

• do not include site_properties keys when values are missing/incomplete (avoid None lists), or
• provide an option to fetch “clean” structures without these optional site properties.

This would prevent common downstream crashes (ASE conversion is one example).

Version

v0.45.15

Which OS?

• MacOS
• Windows
• Linux

Log output

[tsmathis]

Hi @Sankek, this is a side-effect of MP's storage backend that will persist as long as pymatgen data types are internally used to represent structures (and similar objects).

A workaround for this is present in the deserialization mechanism used for MP's data models, which I had thought was propagated to mp_api but I will have to re-check what is happening there since that is not the case based on your issue.

In the mean time you can either (a) remove those site_properties yourself using the .remove_site_property(...) helper method that exists on pymatgen structures, or (b) redownload the summary collection but defer the deserialization and use the document model for SummaryDoc after the download is complete:

from mp_api.client import MPRester
from emmet.core.summary import SummaryDoc


with MPRester(use_document_model=False, monty_decode=False) as mpr:  # defer deserialization
    docs = mpr.materials.summary.search()

docs = [SummaryDoc(**doc) for doc in docs]  # hydrate objects

# print(docs.[0].structure.site_properties
# {'magmom': [0.0, 0.0, 0.0, -0.0, -0.0, -0.0, -0.0, 0.0, -0.0, -0.0, 0.0, 0.0, -0.0, 0.0, -0.0, -0.0, -0.0, 0.0, -0.0, -0.0, 0.0, 0.0, -0.0, -0.0]}

[Sankek]

Hi @Sankek, this is a side-effect of MP's storage backend that will persist as long as pymatgen data types are internally used to represent structures (and similar objects).

A workaround for this is present in the deserialization mechanism used for MP's data models, which I had thought was propagated to mp_api but I will have to re-check what is happening there since that is not the case based on your issue.

In the mean time you can either (a) remove those site_properties yourself using the .remove_site_property(...) helper method that exists on pymatgen structures, or (b) redownload the summary collection but defer the deserialization and use the document model for SummaryDoc after the download is complete:

from mp_api.client import MPRester
from emmet.core.summary import SummaryDoc

with MPRester(use_document_model=False, monty_decode=False) as mpr: # defer deserialization
docs = mpr.materials.summary.search()

docs = [SummaryDoc(**doc) for doc in docs] # hydrate objects

print(docs.[0].structure.site_properties

{'magmom': [0.0, 0.0, 0.0, -0.0, -0.0, -0.0, -0.0, 0.0, -0.0, -0.0, 0.0, 0.0, -0.0, 0.0, -0.0, -0.0, -0.0, 0.0, -0.0, -0.0, 0.0, 0.0, -0.0, -0.0]}

Your workaround fixes it for me.
Using MPRester(use_document_model=False, monty_decode=False) and then SummaryDoc(**doc) removes the None site_properties and AseAtomsAdaptor.get_atoms() works. Thanks!

[esoteric-ephemera]

Hey @Sankek this should be fixed in mp_api==0.46.0:

from mp_api.client import MPRester

with MPRester() as mpr:
    mp_docs = mpr.materials.summary.search()
atoms = {doc.material_id.string : doc.structure.to_ase_atoms() for doc in mp_docs}

(You don't need the AseAtomsAdaptor explicitly.) Can you try pip install --upgrade mp_api==0.46.0 and see if it's working for you?

[Sankek]

Upgraded to mp_api==0.46.0, it works now. Thanks!