[READ-ONLY] Mirror of https://github.com/FoxxMD/multi-scrobbler. Scrobble plays from multiple sources to multiple clients
docs.multi-scrobbler.app
deezer
docker
jellyfin
koito
lastfm
listenbrainz
maloja
mopidy
mpris
music
music-assistant
plex
scrobble
self-hosted
spotify
subsonic
tautulli
youtube-music
feat: Improved retention policy
* Add ENVs for compacting
* Default compacting all data types
* Add rention guidance to docs
+278
-10
+267
docsite/docs/configuration/configuration.mdx
+267
docsite/docs/configuration/configuration.mdx
···
442
442
443
443
</Tabs>
444
444
445
+
## Database
446
+
447
+
Multi-scrobbler depends on a SQLite database (`ms.db`) that is created on first run and stored in the [`CONFIG_DIR`](/installation/?dockerSetting=storage#recommended-settings). When upgrading Multi-scrobbler version, if there are any required database changes than this database is [automatically backed up and migrated.](/updating#database)
448
+
449
+
The database stores *all* Plays for your Sources/Clients as well as metadata and debugging information to help troubleshoot issues. Each Play is associated with a Source/Config in the database based on your configuration.
450
+
451
+
You **should set IDs for each Source/Client** so that the database can identify these even when the configuration is changed.
452
+
453
+
### Retention
454
+
455
+
<DetailsAdmo status="note" summary="How much storage does the database use?">
456
+
457
+
The amount of data stored for each Play can widely vary based on a few factors:
458
+
459
+
* how much data the Source service exposes
460
+
* how many clients each Source is scrobble to (Plays are duplicated for each Client a Source sends a scrobble to)
461
+
* if you are using any [Transforms](/configuration/transforms) the diff of each step is stored, along with any external request/response data (like [Musicbrainz queries](/configuration/transforms/musicbrainz))
462
+
463
+
The [MS repository contains a benchmark](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/backend/tests/database/drizzle.test.ts#L551) to measure an average database size in a few common scenarios.
464
+
465
+
<Tabs groupId="dbSize">
466
+
467
+
<TabItem label="Minimal" value="minimal">
468
+
469
+
Assuming your Sources send a minimal amount of data or you have compacted all plays:
470
+
471
+
| Play Count | DB Size |
472
+
| ---------- | -------- |
473
+
| 100 | `160kb` |
474
+
| 1000 | `1MB` |
475
+
| 10k | `10.2MB` |
476
+
477
+
</TabItem>
478
+
479
+
<TabItem label="With Original Input" value="input">
480
+
481
+
Assuming your Sources have a non-trivial amount of input data (like Spotify or Listenbrainz) that is not compacted:
482
+
483
+
| Play Count | DB Size |
484
+
| ---------- | -------- |
485
+
| 100 | `356kb` |
486
+
| 1000 | `3MB` |
487
+
| 10k | `29.6MB` |
488
+
489
+
</TabItem>
490
+
491
+
<TabItem label="All Debug Deta" value="all">
492
+
493
+
Assuming your Sources/Clients:
494
+
495
+
* have a non-trivial amount of input data (like Spotify or Listenbrainz)
496
+
* and has many [Transforms](/configuration/transforms) steps that include requests
497
+
* nothing is compacted
498
+
499
+
| Play Count | DB Size |
500
+
| ---------- | -------- |
501
+
| 100 | `684kb` |
502
+
| 1000 | `6.28MB` |
503
+
| 10k | `61.3MB` |
504
+
505
+
</TabItem>
506
+
507
+
</Tabs>
508
+
509
+
</DetailsAdmo>
510
+
511
+
A retention policy can be configured to delete Plays, or unused debug data, from the database after a certain amount of time. If no configuration is provided then a default policy is used that should be reasonable for most users.
512
+
513
+
<Tabs groupId="policy" queryString>
514
+
515
+
<TabItem label="Compaction" value="compaction">
516
+
517
+
The **Compaction** Retention Policy is used to delete different types of debug data from your stored Plays.
518
+
519
+
This is a useful way to reduce used storage space when you are not having problems with your Plays, or iterating on a configuration, that requires referencing all this extra data.
520
+
521
+
There are two types of data that can be compacted (deleted from the Play):
522
+
523
+
* `input` - this is the untouched data retrieved by Multi-scrobbler, from a Source, and used to generate a Play/scrobble. This can be used to reconstruct and replay a Play, when used from troubleshooting or reporting an issue
524
+
* `transform` - this is all of the steps generated by [transforms](/configuration/transforms), the diff of the Play resulting from the step, and any request/responses used to complete the step
525
+
526
+
:::note[Defaults]
527
+
528
+
When no Compact configuration is provided, Multi-scrobbler uses this policy:
529
+
530
+
* Compact (delete) `input` and `transform` data on all Plays after 3 days
531
+
532
+
:::
533
+
534
+
#### Configuring Compaction Policy
535
+
536
+
<details>
537
+
538
+
<summary>Details</summary>
539
+
540
+
Each value in the configuration properties below can be either
541
+
542
+
* a number of seconds EX `3600` = 10 minutes
543
+
* a unit of a common duration with the pattern `X unit` EX
544
+
* `30 minutes`
545
+
* `5 hours`
546
+
* `2 days`
547
+
548
+
<Tabs groupId="configType" queryString>
549
+
550
+
<TabItem value="env" label="ENV">
551
+
552
+
* `COMPACT_PROPERTIES` - which properties to compact
553
+
* `RETENTION_COMPACT_AFTER` - Default to use for all Plays
554
+
* `RETENTION_COMPACT_COMPLETED_AFTER` - Compact only completed Plays after...
555
+
* `RETENTION_COMPACT_FAILED_AFTER` - Compact only failed Plays after...
556
+
* `RETENTION_COMPACT_DUPED_AFTER` - Compact only duped/discard Plays after...
557
+
558
+
Example
559
+
560
+
```ini
561
+
# only delete input when compacting
562
+
COMPACT_PROPERTIES=input
563
+
# compact all plays after 3 days
564
+
RETENTION_COMPACT_AFTER=3 days
565
+
# specifically compact completed plays after 30 minutes
566
+
RETENTION_COMPACT_COMPLETED_AFTER=30 minutes
567
+
```
568
+
569
+
</TabItem>
570
+
<TabItem value="aio" label="AIO">
571
+
572
+
Compacting all Play types and deleting both input and transform:
573
+
574
+
```json title="config.json"
575
+
{
576
+
"database": {
577
+
"retention": {
578
+
"compactAfter": "3 days",
579
+
"compact": [
580
+
"input",
581
+
"transform"
582
+
]
583
+
}
584
+
}
585
+
}
586
+
```
587
+
588
+
* Delete only input during compacting
589
+
* Compact all after 3 days except completed which compacts after 30 minutes
590
+
591
+
```json title="config.json"
592
+
{
593
+
"database": {
594
+
"retention": {
595
+
"compactAfter": {
596
+
"completed": "30 minutes",
597
+
"duped": "3 days",
598
+
"failed": "3 days"
599
+
},
600
+
"compact": [
601
+
"input"
602
+
]
603
+
}
604
+
}
605
+
}
606
+
```
607
+
608
+
</TabItem>
609
+
</Tabs>
610
+
611
+
</details>
612
+
613
+
</TabItem>
614
+
615
+
<TabItem label="Deletion" value="deletion">
616
+
617
+
The **Deletion** Retention Policy is used to delete different types of stored Plays from Multi-scrobbler database.
618
+
619
+
**This does not delete Plays from your Clients.** It's only deleting the "in-flight" data MS used to create the scrobble that was eventually sent to your clients.
620
+
621
+
:::warning[Plays Should be Ephemeral]
622
+
623
+
**Multi-Scrobbler is not designed to store Plays/Scrobbles indefinitely.**
624
+
625
+
It should scale fine for thousands of scrobbles but it not meant to store 10's of thousands of scrobbles forever. It is not a scrobbler server.
626
+
627
+
You **should** set a reasonable deletion policy so that MS stores less than 1000 scrobbles at a time, ideally less.
628
+
629
+
:::
630
+
631
+
:::note[Defaults]
632
+
633
+
When no Deletion policy configuration is provided, Multi-scrobbler uses this policy:
634
+
635
+
* Delete all Plays after 7 days
636
+
637
+
:::
638
+
639
+
#### Configuring Deletion Policy
640
+
641
+
<details>
642
+
643
+
<summary>Details</summary>
644
+
645
+
Each value in the configuration properties below can be either
646
+
647
+
* a number of seconds EX `3600` = 10 minutes
648
+
* a unit of a common duration with the pattern `X unit` EX
649
+
* `30 minutes`
650
+
* `5 hours`
651
+
* `2 days`
652
+
653
+
<Tabs groupId="configType" queryString>
654
+
655
+
<TabItem value="env" label="ENV">
656
+
657
+
* `RETENTION_DELETE_AFTER` - Default to use for all Plays
658
+
* `RETENTION_DELETE_COMPLETED_AFTER` - Delete only completed Plays after...
659
+
* `RETENTION_DELETE_FAILED_AFTER` - Delete only failed Plays after...
660
+
* `RETENTION_DELETE_DUPED_AFTER` - Delete only duped/discard Plays after...
661
+
662
+
Example
663
+
664
+
```ini
665
+
# delete all plays after 3 days
666
+
RETENTION_DELETE_AFTER=3 days
667
+
# specifically, delete completed plays after 30 minutes
668
+
RETENTIOND_DELETE_COMPLETED_AFTER=30 minutes
669
+
```
670
+
671
+
</TabItem>
672
+
<TabItem value="aio" label="AIO">
673
+
674
+
Deleting all Play types after 3 days:
675
+
676
+
```json title="config.json"
677
+
{
678
+
"database": {
679
+
"retention": {
680
+
"deleteAfter": "3 days"
681
+
}
682
+
}
683
+
}
684
+
```
685
+
686
+
* Delete all after 3 days except completed which are deleted after 30 minutes
687
+
688
+
```json title="config.json"
689
+
{
690
+
"database": {
691
+
"retention": {
692
+
"deleteAfter": {
693
+
"completed": "30 minutes",
694
+
"duped": "3 days",
695
+
"failed": "3 days"
696
+
},
697
+
}
698
+
}
699
+
}
700
+
```
701
+
702
+
</TabItem>
703
+
</Tabs>
704
+
705
+
</details>
706
+
707
+
</TabItem>
708
+
709
+
710
+
</Tabs>
711
+
445
712
## Debug Mode
446
713
447
714
Turning on Debug Mode will
+1
-1
src/backend/common/AbstractComponent.ts
+1
-1
src/backend/common/AbstractComponent.ts
···
58
58
super(config);
59
59
this.transformManager = config.transformManager ?? getRoot().items.transformerManager;
60
60
this.cache = getRoot().items.cache();
61
-
const cProps = config.options?.retention?.compact ?? parseArrayFromMaybeString(process.env.COMPACT_PROPERTIES, {lower: true});
61
+
const cProps = config.options?.retention?.compact ?? parseArrayFromMaybeString(process.env.COMPACT_PROPERTIES ?? 'input,transform', {lower: true});
62
62
if(!cProps.every(isCompactableProperty)) {
63
63
throw new SimpleError(`Compactable properties must be one of 'transform' or 'input'. Given: ${cProps.join(',')}`);
64
64
}
+8
-8
src/backend/common/database/Database.ts
+8
-8
src/backend/common/database/Database.ts
···
4
4
import { childLogger, Logger } from '@foxxmd/logging';
5
5
import { loggerNoop } from '../MaybeLogger.js';
6
6
import { fileExists, fileOrDirectoryIsWriteable } from '../../utils/FSUtils.js';
7
-
import { COMPACTABLE, compactableProperties, CompactableProperty, DEFAULT_RETENTION_DELETE_AFTER, RententionGranular, RetentionConfig, RetentionConfigValue, RetentionOption, RetentionValue, RetentionValueUnparsed } from '../infrastructure/config/database.js';
7
+
import { COMPACTABLE, compactableProperties, CompactableProperty, DEFAULT_RETENTION_COMPACT_AFTER, DEFAULT_RETENTION_DELETE_AFTER, RententionGranular, RetentionConfig, RetentionConfigValue, RetentionOption, RetentionValue, RetentionValueUnparsed } from '../infrastructure/config/database.js';
8
8
import { DurationValue } from '../infrastructure/Atomic.js';
9
9
import { Duration } from 'dayjs/plugin/duration.js';
10
10
import dayjs from 'dayjs';
···
71
71
throw new SimpleError('retention value be of one: false, number, or string');
72
72
}
73
73
74
-
const parseRetentionFromEnv = (): RetentionOption<RetentionValue> => {
75
-
const deleteAfterEnv = process.env.RETENTION_DELETE_AFTER ?? DEFAULT_RETENTION_DELETE_AFTER,
76
-
deleteCompletedEnv = process.env.RETENTION_DELETE_COMPLETED_AFTER ?? deleteAfterEnv,
77
-
deleteFailedEnv = process.env.RETENTION_DELETE_FAILED_AFTER ?? deleteAfterEnv,
78
-
deleteDupedEnv = process.env.RETENTION_DELETE_DUPED_AFTER ?? deleteAfterEnv;
74
+
const parseRetentionFromEnv = (type: string, defaultVal: number = DEFAULT_RETENTION_DELETE_AFTER): RetentionOption<RetentionValue> => {
75
+
const deleteAfterEnv = process.env[`RETENTION_${type}_AFTER`] ?? defaultVal,
76
+
deleteCompletedEnv = process.env[`RETENTION_${type}_COMPLETED_AFTER`] ?? deleteAfterEnv,
77
+
deleteFailedEnv = process.env[`RETENTION_${type}_FAILED_AFTER`] ?? deleteAfterEnv,
78
+
deleteDupedEnv = process.env[`RETENTION_${type}_DUPED_AFTER`] ?? deleteAfterEnv;
79
79
80
80
return {
81
81
completed: parseRetentionValue(deleteCompletedEnv),
···
95
95
96
96
export const getRetentionDeleteAfterFromEnv = () => {
97
97
if (retentionDeleteAfterFromEnv === undefined) {
98
-
const deleteEnv = parseRetentionFromEnv();
98
+
const deleteEnv = parseRetentionFromEnv('DELETE');
99
99
if(isRetentionOptionDurations(deleteEnv)) {
100
100
retentionDeleteAfterFromEnv = deleteEnv;
101
101
} else {
···
106
106
}
107
107
export const getRetentionCompactAfterFromEnv = () => {
108
108
if (retentionCompactAfterFromEnv === undefined) {
109
-
const compactEnv = parseRetentionFromEnv();
109
+
const compactEnv = parseRetentionFromEnv('COMPACT', DEFAULT_RETENTION_COMPACT_AFTER);
110
110
retentionCompactAfterFromEnv = compactEnv;
111
111
}
112
112
return retentionCompactAfterFromEnv;
+2
-1
src/backend/common/infrastructure/config/database.ts
+2
-1
src/backend/common/infrastructure/config/database.ts