Reduce attributes on FAT file system with Mac OS

[floesche]

SD cards generated with Mac OS seem to fail with the index based access to files. This is due to the ._* files generated by Mac OS to keep file attributes. This code remove attributes from the *.pat files before writing to FAT32. Potentially this avoids the ._*.pat files being generated on the SD card.

This PR need thorough testing before considering at merge to main.

[mbreiser]

Local Test Results (Mac, MATLAB R2025b)

Tested the xattr -c approach on a real SD card with 16 reference patterns (patterns/reference/G41_2x12_cw/).

Result: xattr -c does NOT prevent ._* files on FAT32.

Key Finding

After xattr -c runs on the staging files, com.apple.provenance persists:

$ xattr -c pat0001.pat
$ xattr pat0001.pat
com.apple.provenance

This is a macOS security attribute that cannot be removed by xattr -c. When MATLAB's copyfile() writes these files to FAT32, macOS creates ._* AppleDouble files to store the provenance metadata.

Test Output

• ✅ 16 patterns copied, all byte-exact
• ❌ 16 ._* files created in /patterns/ (one per pattern)
• ❌ 3 ._* files created in root (._MANIFEST.bin, ._MANIFEST.txt, ._patterns)
• ❌ Verification failed: found 32 files instead of 16

Recommendation

The current cleanup approach on main (deleting ._* files after copy) is the correct strategy. xattr -c is insufficient because com.apple.provenance is undeletable.

The real question remains: does removing ._* files (but leaving .Spotlight-V100 in root) allow the G4.1 controller to work with Mac-prepared cards?

[floesche]

If removing attributes doesn't work, we should avoid creating SD cards with Mac for now. There will always be the attributes files on the Mac generated FAT32 card and they still be temporally interleaved with the pattern files.

The "recommendation" doesn't make sense in our context. Keeping the attribute files or keeping them does not make a difference is the file access is based on the index of the FAT entry.

Once the file name based access on the teensy is fast enough, it also won't matter what other files are on the card, so removing the attribute files will also be unnecessary.

I suggest closing this PR and using Windows or Linux to create the SD cards.

[floesche]

One thing that could work: on Mac (and only there) create pattern files names that are in the 6.3 name format. Then the attribute files will follow the 8.3 format. Together the pattern file and the associated attribute file might occupy as much space in the FAT index as the long form file names generated in Linux and windows.

But this makes a bad workaround (avoiding file names since that broke directories with more that 50 pattern files) worse. Instead I suggest to focus on getting the file name based access to work in the teensy with the state machine running.

[floesche]

One thing that could work: on Mac (and only there) create pattern files names that are in the 6.3 name format. Then the attribute files will follow the 8.3 format. Together the pattern file and the associated attribute file might occupy as much space in the FAT index as the long form file names generated in Linux and windows.

Even that might not work, depending on the Mac implementation: "." cannot be part of a short filename entry, which means the attribute files might always occupy a "long" entry (see page 33 and following at https://cscie92.dce.harvard.edu/fall2025/slides/FAT32%20File%20Structure.pdf)

[mbreiser]

Claude thought about this some more and found one potential solution! I will test tomorrow:

---

macOS AppleDouble (._*) files on FAT32 — what works and what doesn't

On macOS, every file copied to a FAT32 SD card gets a companion ._* AppleDouble file (because macOS adds com.apple.provenance xattr at the kernel level, and FAT32 can't store extended attributes natively). These extra files occupy FAT32 directory entry slots and shift the dirIndex, which may confuse the G4.1 controller firmware.

Prevention approaches tested (ALL FAILED):

Method	Result
xattr -c before copy (PR #21)	com.apple.provenance reappears immediately
COPYFILE_DISABLE=1 cp	Still creates ._* files
cp -X (no xattr flag)	Still creates ._* files
cat source > dest	Still creates ._* files
dd if=source of=dest	Still creates ._* files
ditto --norsrc	Still creates ._* files
Python shutil.copy	Still creates ._* files

The com.apple.provenance xattr is added at the VFS layer by the kernel — no userspace copy method can avoid it.

Cleanup approach that works: dot_clean -m /Volumes/PATSD/patterns — Apple's own utility for merging/removing AppleDouble files. Reliably removes all ._* files after copy. Tested with 16 reference patterns on a real SD card: zero ._* files remain, all patterns byte-identical (MD5 verified), FAT32 dirIndex shows clean pat0001-pat0016 ordering.

Updated prepare_sd_card_crossplatform.m to use dot_clean -m with a manual-deletion fallback. 13/13 MATLAB tests pass.

Still needs: G4.1 controller lab test with a Mac-prepared card to confirm Frank's hypothesis that only /patterns/ dirIndex matters (i.e., .Spotlight-V100 in root is harmless).

---

[floesche]

From the documentation I found it looks like dot_clean -m merges the attributes to the file, then unlinks the AppleDouble file. In relevant parts, that's the same logic that your current MATLAB code does and will lead to the same result (extra index entries in FATnthat are marked as deleted). dot_clean -m does not seem to run low level fat reindexing operation, which would be needed. So it would not fix the problem after copying files to the SD card.

If you could merge the attributes on your internal drive before copying the files (basically replacing my xattr -c) and your Mac doesn't recreate the (provenance or other) attribute immediately of while copying the files, that could work.

And we already confirmed this (also for windows and Linux hidden directories and files):

Still needs: G4.1 controller lab test with a Mac-prepared card to confirm Frank's hypothesis that only /patterns/ dirIndex matters (i.e., .Spotlight-V100 in root is harmless).

[floesche]

What could work: run the current code from main that removes AppleDouble or run the dot_clean on the SD card. Once the ._ files are removed, run fatsort [1] on the SD card's patterns/.

[1] https://fatsort.sourceforge.io/, probably available through homebrew

[mbreiser]

Lab Test Results (Mar 5)

Tested two Mac-prepared SD cards on the G4.1 controller — both failed:

Test 1: dot_clean -m only

• Freshly formatted card (diskutil eraseDisk FAT32 PATSD MBRFormat)
• Spotlight disabled (mdutil -d + mdutil -i off)
• 16 reference patterns copied to /patterns/
• dot_clean -m removed all 19 ._* files (16 in /patterns/, 3 in root)
• .fseventsd removed
• Result: Controller failed to load patterns

Test 2: dot_clean -m + fatsort (Frank's suggestion)

• Same card as Test 1
• Unmounted, ran sudo fatsort -a /dev/disk4s1 to compact/sort FAT32 directory entries
• Remounted and verified: 16 .pat files in clean dirIndex order, 0 ._* files
• Result: Controller still failed

Full list of approaches tested

Method	Purpose	Result
xattr -c before copy (PR #21)	Prevent ._* files	❌ com.apple.provenance is kernel-level, persists
COPYFILE_DISABLE=1 cp	Prevent ._* files	❌ Still creates ._*
cp -X	Prevent ._* files	❌ Still creates ._*
cat source > dest	Prevent ._* files	❌ Still creates ._*
dd if=source of=dest	Prevent ._* files	❌ Still creates ._*
ditto --norsrc	Prevent ._* files	❌ Still creates ._*
Python shutil.copy	Prevent ._* files	❌ Still creates ._*
dot_clean -m post-copy	Remove ._* files	✅ Removes files, but ❌ controller still fails
dot_clean -m + fatsort	Remove ._* + compact FAT32 dir	✅ Clean dirIndex, but ❌ controller still fails

Conclusion

The G4.1 controller failure with Mac-prepared cards goes beyond ._* files and FAT32 dirIndex ordering. Something else about macOS-formatted FAT32 is incompatible with the firmware — possibly .Spotlight-V100 (which macOS locks and cannot be deleted while mounted), or differences in how macOS newfs_msdos creates the FAT32 filesystem compared to Windows format.

Recommendation: Close this PR. Use Windows or Linux for SD card creation. Long-term fix: Teensy firmware with filename-based pattern access.