[READ-ONLY] Mirror of https://github.com/FoxxMD/multi-scrobbler. Scrobble plays from multiple sources to multiple clients
docs.multi-scrobbler.app
deezer
docker
jellyfin
koito
lastfm
listenbrainz
maloja
mopidy
mpris
music
music-assistant
plex
scrobble
self-hosted
spotify
subsonic
tautulli
youtube-music
Merge branch 'storybook'
+166
-43
+65
docsite/docs/configuration/clients/clients.mdx
+65
docsite/docs/configuration/clients/clients.mdx
···
23
23
24
24
## Features
25
25
26
+
### Duplicate Detection
27
+
28
+
Before a Play is scrobbled to a client MS checks existing scrobbles from the Client's API to see if the Play has already been scrobbled.
29
+
30
+
For each Play, MS fetches (cached) scrobbles from the Client in a time range inclusive of the Play's listening timestamp and then scores existing scrobbles against the Play based on:
31
+
32
+
* Similarity of Title, Artists, and Album
33
+
* Temporal closeness of the Play's timestamp to the existing scrobble's timestamp
34
+
* Whether MS detected the Play as a repeat (applicable to Sources that report realtime Player data)
35
+
36
+
Detailed scoring breakdowns against each existing scrobble are logged at the `TRACE` level.
37
+
38
+
<details>
39
+
40
+
<summary>Detailed Explanation</summary>
41
+
42
+
A **candidate** (to be scrobbled) Play is first transformed using the configured [`compare.candidate` Hook](/configuration/transforms/#hook), if any exists.
43
+
44
+
Next, MS checks (up to) the last 100 scrobbles *in-memory* scrobbles that it has made. These are *not* from the Client but the actual scrobbles MS made while it has been running. The data from these scrobbles is much richer than what is usually parsed from the Client which makes it easier to detect duplicates from.
45
+
46
+
If no in-memory scrobble matches then MS starts comparing the candidate against historical scrobbles fetched from an inclusive time range of the candidate's timestamp.
47
+
48
+
<h3>Matching Title/Artist/Album</h3>
49
+
50
+
For these string-based values MS uses an [token-order-invariant](https://github.com/FoxxMD/multi-scrobbler/blob/0dfa4c7aad6df98e13aee6d395827665c2414adb/src/backend/utils/StringUtils.ts#L299) method that scores string similarity based on a (token count) weighted average of two [similarity](https://foxxmd.github.io/string-sameness/#md:strategies) algorithms, [Levenshtien Distance](https://en.wikipedia.org/wiki/Levenshtein_distance) and [Dice's Coefficient](https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient). The weighted average ensures that very long strings (Track titles) are scored with a confidence proportional to their length.
51
+
52
+
<h3>Matching Timestamp</h3>
53
+
54
+
Timestamps are scored on how temporally close they are. There are four possible scores with decreasing value:
55
+
56
+
* **Exact** - Timestamps are within 1 second of each other
57
+
* **Close** - Timestamps are within `threshold` seconds of each other
58
+
* This is determined by the smallest update interval of Source
59
+
* Some Sources (subsonic) only update every 60 seconds so this is the smallest "close" value possible. Most are 10 seconds. Signified by `(Needed <10s)` in the breakdown example below.
60
+
* **Fuzzy** - One timestamp is within `threshold` seconds of the *end* of the other timestamp
61
+
* Sources can set the scrobble timestamp at different times. Some do it when the track is started listening to, some when it ends, some when the *player* stops.
62
+
* Where possible, MS knows and keeps track of when this timestamp *should* be, for each Source. If it's not possible then Fuzzy may be allowed.
63
+
* **None** - There is no correlation between timestamps
64
+
65
+
<h3>Scoring and Breakdows</h3>
66
+
67
+
A candidate Play must score >= 1 to be detected as a duplicate of an existing scrobble.
68
+
69
+
Each score and a breakdown of the scores for its individual components can be see at the `TRACE` logging level or in the [debug data](/help#copy-play-debug-data) for a scrobble. An example:
70
+
71
+
```
72
+
* Artist: 0.06 * 0.3 = 0.02
73
+
* Title: 0.04 * 0.4 = 0.02
74
+
* Time: (Exact) 1 * 0.5 = 0.50
75
+
* Existing: 19:13:33-04:00 - Candidate: 19:13:33-04:00
76
+
* Temporal Sameness: Exact
77
+
* Play Diff: 0s (Needed <10s)
78
+
* Range Comparison N/A
79
+
Score 0.54 => No Match
80
+
```
81
+
82
+
In each component equation, the first number is the similarity (or temporal closeness).
83
+
84
+
* 0 = no correlation
85
+
* 1 = exactly the same
86
+
87
+
The second number is the *weight* of that component in the final score.
88
+
89
+
</details>
90
+
26
91
### Dead Scrobbles
27
92
28
93
If multi-scrobbler is unable to submit a scrobble to a Client then it places the scrobble into a queue which is retried every 5 minutes for a number of times before it gives up.
+3
src/backend/common/transforms/AbstractTransformer.ts
+3
src/backend/common/transforms/AbstractTransformer.ts
···
9
9
import { playContentInvariantTransform } from "../../utils/PlayComparisonUtils.js";
10
10
import { isSimpleError, SkipTransformStageError, StagePrerequisiteError } from "../errors/MSErrors.js";
11
11
import { capitalize } from "../../../core/StringUtils.js";
12
+
import { StaggerOptions } from "../../utils/AsyncUtils.js";
12
13
13
14
export interface TransformerOptions {
14
15
logger: Logger
···
33
34
cache: Cacheable;
34
35
35
36
name: string;
37
+
38
+
public staggerOpts: Partial<StaggerOptions> = { initialInterval: 0, maxRandomStagger: 0};
36
39
37
40
public constructor(config: TransformerCommon, options: TransformerOptions) {
38
41
super(config);
+4
src/backend/common/transforms/MusicbrainzTransformer.ts
+4
src/backend/common/transforms/MusicbrainzTransformer.ts
···
355
355
public constructor(config: MusicbrainzTransformerConfig, options: TransformerOptions & {clientCache?: Cacheable}) {
356
356
super(config, options);
357
357
this.clientCache = options.clientCache;
358
+
this.staggerOpts = {
359
+
initialInterval: 300,
360
+
maxRandomStagger: 300
361
+
}
358
362
}
359
363
360
364
protected async doBuildInitData(): Promise<true | string | undefined> {
+6
-22
src/backend/common/transforms/TransformerManager.ts
+6
-22
src/backend/common/transforms/TransformerManager.ts
···
110
110
return this.transformers.has(type);
111
111
}
112
112
113
-
protected getTransformerByStage(data: StageConfig): AbstractTransformer {
113
+
public getTransformerType(type: string): AbstractTransformer[] | undefined {
114
+
return this.transformers.get(type);
115
+
}
116
+
117
+
public getTransformerByStage(data: StageConfig): AbstractTransformer {
114
118
const list = this.transformers.get(data.type);
115
119
if (list === undefined || list.length === 0) {
116
120
throw new Error(`No transformer of type '${data.type}' is registered.`);
···
143
147
}
144
148
145
149
public async handleStage(data: StageConfig, play: PlayObject, asyncId: string = nanoid(6)): Promise<[PlayObject, string]> {
146
-
const list = this.transformers.get(data.type);
147
-
if (list === undefined || list.length === 0) {
148
-
throw new Error(`No transformer of type '${data.type}' is registered.`);
149
-
}
150
-
151
-
let t: AbstractTransformer;
152
-
if (list.length > 1) {
153
-
if(data.name === undefined) {
154
-
this.logger.warn(`More than one '${data.type}' transformer but name was not specified, using first registered`);
155
-
t = list[0];
156
-
} else {
157
-
const named = list.find(x => x.name === data.name);
158
-
if(named === undefined) {
159
-
throw new Error(`No ${data.type} transformer with name '${data.name}'`)
160
-
}
161
-
t = named;
162
-
}
163
-
} else {
164
-
t = list[0];
165
-
}
166
-
150
+
const t: AbstractTransformer = this.getTransformerByStage(data);
167
151
try {
168
152
const transformedPlay = await this.asyncStore.run(asyncId, async () => {
169
153
return await t.handle(data, play);
+1
-4
src/backend/ioc.ts
+1
-4
src/backend/ioc.ts
···
27
27
cache?: CacheConfigOptions | MSCache | (() => MSCache)
28
28
mbMap?: MusicBrainzSingletonMap | (() => MusicBrainzSingletonMap)
29
29
transformers?: TransformerCommonConfig[]
30
-
staggerOptions?: Partial<StaggerOptions>
31
30
}
32
31
33
32
const discovered = new prom.Counter({
···
61
60
logger,
62
61
cache,
63
62
mbMap,
64
-
transformers = [],
65
-
staggerOptions,
63
+
transformers = []
66
64
} = options || {};
67
65
const configDir = process.env.CONFIG_DIR || path.resolve(projectDir, `./config`);
68
66
let disableWeb = dw;
···
150
148
cache: () => maybeSingletonCache !== undefined ? () => maybeSingletonCache : cacheFunc,
151
149
mbMap: () => maybeSingletonMb !== undefined ? () => maybeSingletonMb : mbFunc,
152
150
coverArtApi,
153
-
staggerOptions: staggerOptions ?? {},
154
151
}).add((items) => {
155
152
const localUrl = generateBaseURL(baseUrl, items.port)
156
153
return {
+42
-5
src/backend/scrobblers/AbstractScrobbleClient.ts
+42
-5
src/backend/scrobblers/AbstractScrobbleClient.ts
···
125
125
protected problemGauge: Gauge;
126
126
127
127
protected staggerOpts: Partial<StaggerOptions>;
128
+
protected staggerMappers = {
129
+
preCompare: staggerMapper<PlayObject, PlayObject>({concurrency: 2}),
130
+
existing: staggerMapper<ScrobbledPlayObject, ScrobbledPlayObject>({concurrency: 2})
131
+
}
128
132
129
133
constructor(type: any, name: any, config: CommonClientConfig, notifier: Notifiers, emitter: EventEmitter, logger: Logger) {
130
134
super(config);
···
190
194
existingSubmitted: this.findExistingSubmittedPlayObj
191
195
}
192
196
this.existingScrobble = (playObjPre: PlayObject, existingScrobbles: PlayObject[], log?: boolean) => existingScrobble(playObjPre, existingScrobbles, existingScrobbleOpts, log);
193
-
this.staggerOpts = getRoot().items.staggerOptions;
197
+
}
198
+
199
+
protected async postCache(): Promise<void> {
200
+
await super.postCache();
201
+
this.generateStaggerMappers();
202
+
}
203
+
204
+
protected generateStaggerMappers() {
205
+
const {
206
+
preCompare = [],
207
+
compare: {
208
+
existing = []
209
+
} = {}
210
+
} = this.transformRules;
211
+
212
+
if(preCompare.length > 0) {
213
+
let pcInits: number[] = [0],
214
+
pcMaxStagger: number[] = [];
215
+
for(const hook of this.transformRules.preCompare) {
216
+
const t = this.transformManager.getTransformerByStage({type: hook.type, name: hook.name});
217
+
pcInits.push(t.staggerOpts?.initialInterval ?? 0);
218
+
pcMaxStagger.push(t.staggerOpts?.maxRandomStagger ?? 0)
219
+
}
220
+
this.staggerMappers.preCompare = staggerMapper<PlayObject, PlayObject>({initialInterval: Math.max(...pcInits), maxRandomStagger: Math.max(...pcMaxStagger), concurrency: 2});
221
+
}
222
+
223
+
if(existing.length > 0) {
224
+
let eInits: number[] = [0],
225
+
eMaxStagger: number[] = [];
226
+
for(const hook of this.transformRules.postCompare) {
227
+
const t = this.transformManager.getTransformerByStage({type: hook.type, name: hook.name});
228
+
eInits.push(t.staggerOpts?.initialInterval ?? 0);
229
+
eMaxStagger.push(t.staggerOpts?.maxRandomStagger ?? 0)
230
+
}
231
+
this.staggerMappers.existing = staggerMapper<ScrobbledPlayObject, ScrobbledPlayObject>({initialInterval: Math.max(...eInits), maxRandomStagger: Math.max(...eMaxStagger), concurrency: 2});
232
+
}
194
233
}
195
234
196
235
protected getIdentifier() {
···
449
488
450
489
const playObj = await this.transformPlay(playObjPre, TRANSFORM_HOOK.candidate);
451
490
452
-
const sm = staggerMapper<ScrobbledPlayObject, ScrobbledPlayObject>({...this.staggerOpts, concurrency: 2});
453
-
const dtInvariantMatches = (await pMap(this.scrobbledPlayObjs.data, sm(async x => ({...x, play: await this.transformPlay(x.play, TRANSFORM_HOOK.existing)})), {concurrency: 2}))
491
+
const dtInvariantMatches = (await pMap(this.scrobbledPlayObjs.data, this.staggerMappers.existing(async x => ({...x, play: await this.transformPlay(x.play, TRANSFORM_HOOK.existing)})), {concurrency: 2}))
454
492
.filter(x => playObjDataMatch(playObj, x.play));
455
493
456
494
if (dtInvariantMatches.length === 0) {
···
851
889
852
890
queueScrobble = async (data: PlayObject | PlayObject[], source: string) => {
853
891
const plays = (Array.isArray(data) ? data : [data]).map(x => ({...x, meta: {...x.meta, seenAt: dayjs()}}));
854
-
const sm = staggerMapper<PlayObject, PlayObject>({...this.staggerOpts, concurrency: 2});
855
-
for await(const play of pMapIterable(plays, sm(async x => await this.transformPlay(x, TRANSFORM_HOOK.preCompare)), {concurrency: 2})) {
892
+
for await(const play of pMapIterable(plays, this.staggerMappers.preCompare(async x => await this.transformPlay(x, TRANSFORM_HOOK.preCompare)), {concurrency: 2})) {
856
893
try {
857
894
const existingQueued = await this.existingScrobble(play, this.queuedScrobbles.map(x => x.play), false);
858
895
// want to be very confident of this
+40
-6
src/backend/sources/AbstractSource.ts
+40
-6
src/backend/sources/AbstractSource.ts
···
94
94
95
95
protected discoveredCounter: Counter;
96
96
97
-
protected staggerOpts: Partial<StaggerOptions>;
97
+
protected staggerMappers = {
98
+
preCompare: staggerMapper<PlayObject, PlayObject>({concurrency: 2}),
99
+
postCompare: staggerMapper<PlayObject, PlayObject>({concurrency: 2})
100
+
}
98
101
99
102
constructor(type: SourceType, name: string, config: SourceConfig, internal: InternalConfig, emitter: EventEmitter) {
100
103
super(config);
···
112
115
this.emitter = emitter;
113
116
114
117
this.discoveredCounter = getRoot().items.sourceMetics.discovered;
115
-
this.staggerOpts = getRoot().items.staggerOptions;
118
+
}
119
+
120
+
protected async postCache(): Promise<void> {
121
+
await super.postCache();
122
+
this.generateStaggerMappers();
123
+
}
124
+
125
+
protected generateStaggerMappers() {
126
+
const {
127
+
preCompare = [],
128
+
postCompare = [],
129
+
} = this.transformRules;
130
+
131
+
if (preCompare.length > 0) {
132
+
let pcInits: number[] = [0],
133
+
pcMaxStagger: number[] = [0];
134
+
for (const hook of this.transformRules.preCompare) {
135
+
const t = this.transformManager.getTransformerByStage({ type: hook.type, name: hook.name });
136
+
pcInits.push(t.staggerOpts?.initialInterval ?? 0);
137
+
pcMaxStagger.push(t.staggerOpts?.maxRandomStagger ?? 0)
138
+
}
139
+
this.staggerMappers.preCompare = staggerMapper<PlayObject, PlayObject>({ initialInterval: Math.max(...pcInits), maxRandomStagger: Math.max(...pcMaxStagger), concurrency: 2 });
140
+
}
141
+
142
+
if (postCompare.length > 0) {
143
+
let postInits: number[] = [0],
144
+
postMaxStagger: number[] = [0];
145
+
for (const hook of this.transformRules.postCompare) {
146
+
const t = this.transformManager.getTransformerByStage({ type: hook.type, name: hook.name });
147
+
postInits.push(t.staggerOpts?.initialInterval ?? 0);
148
+
postMaxStagger.push(t.staggerOpts?.maxRandomStagger ?? 0)
149
+
}
150
+
this.staggerMappers.postCompare = staggerMapper<PlayObject, PlayObject>({ initialInterval: Math.max(...postInits), maxRandomStagger: Math.max(...postMaxStagger), concurrency: 2 });
151
+
}
116
152
}
117
153
118
154
protected getIdentifier() {
···
222
258
discover = async (plays: PlayObject[], options: { checkAll?: boolean, [key: string]: any } = {}): Promise<PlayObject[]> => {
223
259
const newDiscoveredPlays: PlayObject[] = [];
224
260
225
-
const sm = staggerMapper<PlayObject, PlayObject>({...this.staggerOpts, concurrency: 2});
226
-
for await(const play of pMapIterable(plays, sm(async x => await this.transformPlay(x, TRANSFORM_HOOK.preCompare)), {concurrency: 2})) {
261
+
for await(const play of pMapIterable(plays, this.staggerMappers.preCompare(async x => await this.transformPlay(x, TRANSFORM_HOOK.preCompare)), {concurrency: 2})) {
227
262
if(!(await this.alreadyDiscovered(play, options))) {
228
263
this.addPlayToDiscovered(play);
229
264
newDiscoveredPlays.push(play);
···
254
289
return;
255
290
}
256
291
newDiscoveredPlays.sort(sortByOldestPlayDate);
257
-
const sm = staggerMapper<PlayObject, PlayObject>({...this.staggerOpts, concurrency: 2});
258
292
this.emitter.emit('discoveredToScrobble', {
259
-
data: await pMap(newDiscoveredPlays, sm(async (x) => await this.transformPlay(x, TRANSFORM_HOOK.postCompare)), {concurrency: 2}),
293
+
data: await pMap(newDiscoveredPlays, this.staggerMappers.postCompare(async (x) => await this.transformPlay(x, TRANSFORM_HOOK.postCompare)), {concurrency: 2}),
260
294
options: {
261
295
...options,
262
296
checkTime: newDiscoveredPlays[newDiscoveredPlays.length-1].data.playDate.add(2, 'second'),
+2
-2
src/backend/sources/ListenbrainzSource.ts
+2
-2
src/backend/sources/ListenbrainzSource.ts
···
88
88
89
89
getUpstreamRecentlyPlayed = async (options: RecentlyPlayedOptions = {}): Promise<PlayObject[]> => {
90
90
try {
91
-
return await this.getScrobblesForTimeRange({limit: 20});
91
+
return await this.getScrobblesForTimeRange({limit: 20, ...options});
92
92
} catch (e) {
93
93
throw e;
94
94
}
···
98
98
return 'second';
99
99
}
100
100
101
-
protected getBackloggedPlays = async (options: RecentlyPlayedOptions = {}) => await this.getRecentlyPlayed({formatted: true, ...options})
101
+
protected getBackloggedPlays = async (options: RecentlyPlayedOptions = {}) => await this.getUpstreamRecentlyPlayed({formatted: true, ...options})
102
102
103
103
getNewPlayer = (logger: Logger, id: PlayPlatformId, opts: PlayerStateOptions) => new NowPlayingPlayerState(logger, id, opts);
104
104
}
+1
-1
src/backend/tests/setup.ts
+1
-1
src/backend/tests/setup.ts
···
2
2
import { getRoot } from "../ioc.js";
3
3
import { transientCache } from './utils/CacheTestUtils.js';
4
4
5
-
const root = getRoot({cache: transientCache, logger: loggerTest, staggerOptions: {initialInterval: 1, maxRandomStagger: 1}});
5
+
const root = getRoot({cache: transientCache, logger: loggerTest});
6
6
root.items.cache().init();
+2
-2
src/backend/utils/AsyncUtils.ts
+2
-2
src/backend/utils/AsyncUtils.ts
-1
src/backend/utils/ListenFetchUtils.ts
-1
src/backend/utils/ListenFetchUtils.ts
···
32
32
while (more) {
33
33
requestCount++;
34
34
const reqOptsHint: string[] = [
35
-
`Between ${todayAwareFormat(dayjs(currOpts.from))} and ${todayAwareFormat(dayjs(currOpts.to))}`
36
35
];
37
36
if(currOpts.to !== undefined && currOpts.from !== undefined) {
38
37
reqOptsHint.push(`Between ${todayAwareFormat(dayjs(currOpts.from))} and ${todayAwareFormat(dayjs(currOpts.to))}`);