[Feature Request] Track Duplicates with Differing Track IDs

[carlyd95]

Zotify Version
v0.16.9

Bug Description
This is a revisitation of bug #158. It seems you might not have seen my comments after the issue was closed. Unfortunately, it is still causing duplicates in an undesirable way when batch downloading multiple playlists from a file with playlist URLs.

It seems scenario 2 is still an issue for batch playlist download via file when a song is downloaded which has already been downloaded via a different track ID (I hate that this exists in Spotify tbh). Currently a copy is being made for each song that has already been downloaded when that same song is downloaded but from a different track ID.

Scenario 2 (Description: download a track that already has a file saved via a different track ID):

I have already downloaded this track:
https://open.spotify.com/track/7fc4OLUDFGSbfBfwp6gWua (Angels - Armin van Buuren) [Track ID for the single that was first released]

The same track was also released on his album (which I have not downloaded before):
https://open.spotify.com/track/10x4YPpjuHsYhYMdMyH64X

Below are the results of downloading this track (album track ID) on 2 identical playlists:

OD = True:

• 1 copy of the track is made and saved as 'Armin van Buuren - Angels_1.mp3'
• it is logged to .song_ids & .song_archive

OD = False:

• 1 copy of the track is made and saved as 'Armin van Buuren - Angels_1.mp3'
• it is logged to .song_ids & .song_archive

Playlists used:

• TEST
• TEST (2)

I don't know how best this could be worked in with the current features that are in place but ultimately I want to avoid duplicates being saved if the track title and Artists[] are the same. I have come across 1 remix that has the same track title and the same main artist but the artist array has one additional artist added which differentiates it as well beyond the unique track ID.

I hope this makes sense. I appreciate your help! 😄

[Googolplexed0]

Unfortunately, it is still causing duplicates in an undesirable way

This library defines a track's identity using only its UUID. If two tracks have different UUIDs, they are not duplicates, end of discussion. This is my stance mostly for sanity reasons.

1. Metadata is known to be actively changed by artists/labels and often by the source itself. Any filter that looks at metadata to determine "non-UUID duplicates" can spontaneously fail a spot check with no code changes. This would add an entirely new dimension of bug reproducibility concerns. I don't want to be responsible for maintaining code that I can't test or fix reliably.
2. There is a spectrum to what different users would consider a "non-UUID duplicate". To build a solution I would either have to unilaterally decide where that line is drawn (not widely-usable and inconsistent with this library's ethos), try to reach a community consensus (painful and slow, likely impossible), or build tons of options to appease most use cases (feasible but a lot of work).
3. The problem is generally amorphous. Even with specific, agreed criteria to define a "non-UUID duplicate" there would still be near-miss edge cases that would make the filter feel inconsistent. A feature that feels unreliable is unreliable.
4. There are also implementation level concerns. String equality is pretty strict, especially with things like utf-8 lookalike characters, title translations, and localization effects. This becomes exponential when you consider that each Track contains multiple Artist objects. There might be ways around this but they would be computationally expensive at best.
5. Zotify isn't designed to be a music manager. Anyone can use it that way if it works for them and I am happy to add features to improve that use case. But honestly that is outside of the scope of the library.

There are a few more hurdles but writing more here would be a bit much.

TL;DR - I have been thinking about how to handle "non-UUID duplicates" over the years but could never come up with a good solution. So I won't handle them. Sorry. That makes this more of a [Feature Request] that is on the farthest backburner. I won't say it will never be implemented, but don't expect anything in the near future unless something changes.

[DerKO9]

If two tracks have different UUIDs, they are not duplicates, end of discussion.

I have this opinion as well. Ive been thinking of ways to make the identity of the song be a string like "artist - song name" to avoid duplicates but it makes more problems than its worth.

For my own management of all the downloaded songs I consider the given song id to be the single source of truth of what songs I have downloaded or not and I include the id in the file name. Specifically I like using the output: {artist}_{song_name}_{id}. I do my best to make sure the playlists I download are not having different releases of the same song, but ultimately I just shrug my shoulders at the few duplicates out of my control.

[carlyd95]

@Googolplexed0 hey, thanks for your response. All of what you said makes sense. I agree.

I tried checking the hash before conversion but it seems that isn't even consistent.

I will implement a hacky fix and wait for a better fix in the mean time. I may look into fpcalc later and use that if a file already exists.

Thanks for explaining, I appreciate the logic. :D

[carlyd95]

@Googolplexed0

I am looking at a way to handle duplicates before sending tracklist to zotify. And it seems like most duplicates could be handled by comparing ISRC.

Examples:

GENUINE DUPLICATE:

10x4YPpjuHsYhYMdMyH64X
Armin van Buuren, Punctual, EVALINA - Angels
ISRC: NLF712501262

7fc4OLUDFGSbfBfwp6gWua
Armin van Buuren, Punctual, EVALINA - Angels
ISRC: NLF712501262

0QWfgffeGPTSuoPgAzXxYh
Armin van Buuren, Punctual, EVALINA - Angels
ISRC: NLF712501262

SAME TRACK TITLE BUT UNIQUE:

5MGzRBSlXuNhN19ze9o5GY
TECH IT DEEP, Gordo - Maria Maria
ISRC: GBCEN2400227

4wP6mzWZC94dYmU6OYeljf
TECH IT DEEP - Maria Maria
ISRC: NL7ZX2200001

[DerKO9]

I like that idea! Ive been experimenting with the fpcalc/acoustid fingerprinting on songs to detect duplicates and while I think it could work, it would be nice to have an actual reliable key for songs.

I would love an ISRC tag on songs if its possible. Maybe the ability to put "{ISRC}" in the output formatting for a song files name too.

[Googolplexed0]

I am looking at a way to handle duplicates before sending tracklist to zotify. And it seems like most duplicates could be handled by comparing ISRC.

ISRC would actually be a decent measure. I think it is returned by both ways of fetching metadata.

I would love an ISRC tag on songs if its possible. Maybe the ability to put "{ISRC}" in the output formatting for a song files name too.

I will probably implement ISRC as an output-formattable metadata right away but hold off on adding duplicate checking logic with it for a bit. I need to think more about implementation. What I have in mind would probably require an update to the .song_archive formatting (which is something I have wanted to overhaul for a while anyway) but would still be involved.

[DerKO9]

I made a pull request for what I described. I tested it locally with a couple songs and playlists. Feel free to tidy it up if needed, I'm still pretty unfamiliar with the code base so hopefully it follows any conventions you have ok.
#204