foxxmd.dev / string-sameness
[READ-ONLY] Mirror of https://github.com/FoxxMD/string-sameness. Compare the sameness of two strings foxxmd.github.io/string-sameness/
compare cosine-similarity dice-coefficient levenshtein-distance sameness string text typescript
0
Fork 0

Configure Feed

Select the types of activity you want to include in your feed.

docs: Add new features and fix import usage

FoxxMD (Oct 26, 2023, 3:25 PM EDT) 69b0872f 7337c2de

+140 -18
+34 -5
README.md
··· 51 51 52 52 The average of the scores from all passed strategies is returned as `highScore` (and `highScoreWeighted`) from `stringSameness()` 53 53 54 - When no strategies are explicitly passed a default set of strategies is used, found in `@foxxmd/string-sameness/strategies`: 54 + When no strategies are explicitly passed a default set of strategies is used, found in `import {defaultStrategies} from @foxxmd/string-sameness;`: 55 55 56 56 * [Dice's Coefficient](https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) in [`diceSimilarities.ts`](/src/matchingStrategies/diceSimilarity.ts) 57 57 * [Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity) in [`cosineSimilarities.ts`](/src/matchingStrategies/cosineSimilarity.ts) 58 58 * [Levenshtein Distance](https://en.wikipedia.org/wiki/Levenshtein_distance) in [`levenSimilarities.ts`](/src/matchingStrategies/levenSimilarity.ts) 59 + 60 + Strategies can be accessed individually using `import {strategies} from @foxxmd/string-sameness` 59 61 60 62 ### Bring Your Own Strategy 61 63 ··· 107 109 108 110 Pass a list of functions using `{transforms: []}` to transform the strings before comparison. When not explicitly provided a default set of functions is applied to normalize the strings (to remove trivial differences): 109 111 112 + * normalize unicode EX convert Ö => O 110 113 * convert to lowercase 111 114 * trim (remove whitespace at beginning/end) 112 115 * remove non-alphanumeric characters (punctuation and newlines) 113 116 * replace any instances of 2 or more consecutive whitespace with 1 whitespace 114 117 115 - This default set of functions is exported as `defaultStrCompareTransformFuncs`. 118 + * The default set of transformer functions is exported as `import {strDefaultTransforms} from @foxxmd/string-sameness;` 119 + * All built-in transformers can be found at `import {transforms} from @foxxmd/string-sameness;` 116 120 117 121 Example of supplying your own transform functions: 118 122 ··· 128 132 const result = stringSameness('This is one sentence', 'This is another sentence', {transforms: myFuncs}); 129 133 ``` 130 134 135 + ## Token Re-ordering 136 + 137 + If tokens (word) ordering in the strings is not important you can choose to have string-sameness attempt to re-order all words before comparing sameness. This makes comparison scores much closer to "absolute sameness in all characters within string". EX: 138 + 139 + * `this is correct order` 140 + * `order correct this is` 141 + 142 + Scores 60 **without** reordering 143 + 144 + Scores 100 **with** reordering 145 + 146 + Behavior caveats: 147 + 148 + * The **second** string argument is reordered to match the **first** string argument 149 + * If the second string is longer than the first than any non-matched words are concatenated to the end of the re-ordered string in the same order they were found 150 + 151 + To use: 152 + 153 + ```js 154 + import {stringSameness} from '@foxxmd/string-sameness'; 155 + 156 + const res = stringSameness(strA, strB, {reorder: true}); 157 + ``` 158 + 131 159 ## Factory 132 160 133 161 For convenience, a factory function is also provided: 134 162 135 163 ```ts 136 - import {createStringSameness} from "@foxxmd/string-sameness"; 137 - import {levenStrategy} from "@foxxmd/string-sameness/strategies"; 164 + import {createStringSameness, strategies} from "@foxxmd/string-sameness"; 138 165 import {myTransforms, myStrats} from './util'; 139 166 167 + const {levenStrategy} = strategies; 168 + 140 169 // sets the default object to used with the third argument for `stringSameness` 141 - const myCompare = createStringSameness({transforms: myTransforms, strategies: myStrats}); 170 + const myCompare = createStringSameness({transforms: myTransforms, strategies: [levenStrategy, ...myStrats]}); 142 171 143 172 // uses myTransforms and myStrats 144 173 const plainResult = myCompare('This is one sentence', 'This is another sentence');
+12
dist/commonjs/atomic.d.ts
··· 1 1 export interface StringComparisonOptions { 2 + /** 3 + * An array of transformations to apply to each string before comparing similarity 4 + * */ 2 5 transforms?: StringTransformFunc[]; 6 + /** 7 + * An array of strategies used to score similarity. All strategies scores are combined for an average high score. 8 + * */ 3 9 strategies?: ComparisonStrategy<ComparisonStrategyResultValue>[]; 10 + /** 11 + * Reorder second string so its token match order of first string as closely as possible 12 + * 13 + * Useful when only the differences in content are important, but not the order of the content 14 + * */ 15 + reorder?: boolean; 4 16 } 5 17 export interface StringSamenessResult { 6 18 strategies: {
+1
dist/commonjs/index.d.ts
··· 2 2 import { strDefaultTransforms, transforms } from "./normalization/index.js"; 3 3 declare const defaultStrategies: import("./atomic.js").ComparisonStrategy<import("./atomic.js").ComparisonStrategyResultObject>[]; 4 4 declare const stringSameness: (valA: string, valB: string, options?: StringComparisonOptions) => StringSamenessResult; 5 + export declare const reorderStr: (cleanA: string, cleanB: string, options?: StringComparisonOptions) => string; 5 6 declare const createStringSameness: (defaults: StringComparisonOptions) => (valA: string, valB: string, options?: StringComparisonOptions) => StringSamenessResult; 6 7 declare const strategies: { 7 8 diceStrategy: import("./atomic.js").ComparisonStrategy<import("./atomic.js").ComparisonStrategyResultObject>;
+37 -3
dist/commonjs/index.js
··· 1 1 "use strict"; 2 2 Object.defineProperty(exports, "__esModule", { value: true }); 3 - exports.strDefaultTransforms = exports.defaultStrCompareTransformFuncs = exports.transforms = exports.strategies = exports.defaultStrategies = exports.createStringSameness = exports.stringSameness = void 0; 3 + exports.strDefaultTransforms = exports.defaultStrCompareTransformFuncs = exports.transforms = exports.strategies = exports.defaultStrategies = exports.createStringSameness = exports.stringSameness = exports.reorderStr = void 0; 4 4 const index_js_1 = require("./matchingStrategies/index.js"); 5 5 const index_js_2 = require("./normalization/index.js"); 6 6 Object.defineProperty(exports, "strDefaultTransforms", { enumerable: true, get: function () { return index_js_2.strDefaultTransforms; } }); ··· 17 17 ]; 18 18 exports.defaultStrategies = defaultStrategies; 19 19 const stringSameness = (valA, valB, options) => { 20 - const { transforms = index_js_2.strDefaultTransforms, strategies = defaultStrategies, } = options || {}; 20 + const { transforms = index_js_2.strDefaultTransforms, strategies = defaultStrategies, reorder = false, } = options || {}; 21 21 const cleanA = transforms.reduce((acc, curr) => curr(acc), valA); 22 - const cleanB = transforms.reduce((acc, curr) => curr(acc), valB); 22 + let cleanB = transforms.reduce((acc, curr) => curr(acc), valB); 23 23 const shortest = cleanA.length > cleanB.length ? cleanB : cleanA; 24 + if (reorder) { 25 + // we want to ignore order of tokens as much as possible (user does not care about differences in word order, just absolute differences in characters overall) 26 + // so we will reorder cleanB so its tokens match the order or tokens in cleanA as closely as possible 27 + // before we run strategies 28 + cleanB = (0, exports.reorderStr)(cleanA, cleanB); 29 + } 24 30 const stratResults = []; 25 31 for (const strat of strategies) { 26 32 if (strat.isValid !== undefined && !strat.isValid(cleanA, cleanB)) { ··· 54 60 }; 55 61 }; 56 62 exports.stringSameness = stringSameness; 63 + const reorderStr = (cleanA, cleanB, options) => { 64 + // to do the reordering we will use stringSameness with the provided strats to match against each token in cleanA and choose the closest token in cleanB 65 + // and add the end concat any remaining tokens from cleanB to the reordered string 66 + const aTokens = cleanA.split(' '); 67 + const bTokens = cleanB.split(' '); 68 + const orderedCandidateTokens = aTokens.reduce((acc, curr) => { 69 + let highScore = 0; 70 + let highIndex = 0; 71 + let index = 0; 72 + for (const token of acc.remaining) { 73 + const result = stringSameness(curr, token, { ...options, reorder: false }); 74 + if (result.highScore > highScore) { 75 + highScore = result.highScore; 76 + highIndex = index; 77 + } 78 + index++; 79 + } 80 + const splicedRemaining = [...acc.remaining]; 81 + if (highIndex <= splicedRemaining.length - 1) { 82 + splicedRemaining.splice(highIndex, 1); 83 + } 84 + const ordered = highIndex <= acc.remaining.length - 1 ? acc.ordered.concat(acc.remaining[highIndex]) : acc.ordered; 85 + return { ordered: ordered, remaining: splicedRemaining }; 86 + }, { ordered: [], remaining: bTokens }); 87 + const allOrderedCandidateTokens = orderedCandidateTokens.ordered.concat(orderedCandidateTokens.remaining); 88 + return allOrderedCandidateTokens.join(' '); 89 + }; 90 + exports.reorderStr = reorderStr; 57 91 const createStringSameness = (defaults) => { 58 92 return (valA, valB, options = {}) => stringSameness(valA, valB, { ...defaults, ...options }); 59 93 };
+1 -1
dist/commonjs/index.js.map
··· 1 - {"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/index.ts"],"names":[],"mappings":";;;AAAA,4DAAoH;AAQpH,uDAA0E;AAqFtE,qGArFI,+BAAoB,OAqFJ;AAFpB,2FAnF0B,qBAAU,OAmF1B;AAjFd,MAAM,oBAAoB,GAAG,CAAC,MAAc,EAAE,EAAE;IAC5C,oBAAoB;IACpB,4BAA4B;IAC5B,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC;AACzC,CAAC,CAAA;AAED,MAAM,iBAAiB,GAAG;IACtB,uBAAY;IACZ,wBAAa;IACb,yBAAc;CACjB,CAAA;AAqEG,8CAAiB;AAnErB,MAAM,cAAc,GAAG,CAAC,IAAY,EAAE,IAAY,EAAE,OAAiC,EAAwB,EAAE;IAE3G,MAAM,EACF,UAAU,GAAG,+BAAoB,EACjC,UAAU,GAAG,iBAAiB,GACjC,GAAG,OAAO,IAAI,EAAE,CAAC;IAElB,MAAM,MAAM,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,IAAI,CAAC,CAAC;IACjE,MAAM,MAAM,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,IAAI,CAAC,CAAC;IAEjE,MAAM,QAAQ,GAAG,MAAM,CAAC,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC;IAEjE,MAAM,YAAY,GAA0C,EAAE,CAAC;IAE/D,KAAK,MAAM,KAAK,IAAI,UAAU,EAAE;QAC5B,IAAI,KAAK,CAAC,OAAO,KAAK,SAAS,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE;YAC/D,SAAS;SACZ;QACD,MAAM,GAAG,GAAG,KAAK,CAAC,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QAC3C,MAAM,MAAM,GAAG,OAAO,GAAG,KAAK,QAAQ,CAAC,CAAC,CAAC,EAAC,KAAK,EAAE,GAAG,EAAC,CAAC,CAAC,CAAC,GAAG,CAAC;QAC5D,YAAY,CAAC,IAAI,CAAC;YACd,GAAG,MAAM;YACT,IAAI,EAAE,KAAK,CAAC,IAAI;SACnB,CAAC,CAAC;KACN;IAED,mCAAmC;IACnC,MAAM,WAAW,GAAG,oBAAoB,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;IAE1D,qBAAqB;IACrB,MAAM,SAAS,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,GAAG,GAAG,IAAI,CAAC,KAAK,EAAE,CAAC,CAAC,GAAG,YAAY,CAAC,MAAM,CAAC;IAChG,kCAAkC;IAClC,MAAM,iBAAiB,GAAG,SAAS,GAAG,IAAI,CAAC,GAAG,CAAC,WAAW,EAAE,EAAE,CAAC,CAAC;IAChE,MAAM,QAAQ,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC,GAAgD,EAAE,IAAI,EAAE,EAAE;QAC5F,MAAM,EAAC,IAAI,EAAE,KAAK,EAAE,GAAG,IAAI,EAAC,GAAG,IAAI,CAAC;QACpC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG;YACb,GAAG,IAAI;YACP,KAAK;SACR,CAAC;QACF,OAAO,GAAG,CAAC;IACf,CAAC,EAAE,EAAE,CAAC,CAAC;IACP,OAAO;QACH,UAAU,EAAE,QAAQ;QACpB,SAAS;QACT,iBAAiB;KACpB,CAAA;AACL,CAAC,CAAA;AAmBG,wCAAc;AAjBlB,MAAM,oBAAoB,GAAG,CAAC,QAAiC,EAAE,EAAE;IAC/D,OAAO,CAAC,IAAY,EAAE,IAAY,EAAE,UAAmC,EAAE,EAAE,EAAE,CAAC,cAAc,CAAC,IAAI,EAAE,IAAI,EAAE,EAAC,GAAG,QAAQ,EAAE,GAAG,OAAO,EAAC,CAAC,CAAC;AACxI,CAAC,CAAA;AAgBG,oDAAoB;AAdxB,MAAM,UAAU,GAAG;IACf,YAAY,EAAZ,uBAAY;IACZ,aAAa,EAAb,wBAAa;IACb,cAAc,EAAd,yBAAc;IACd,wBAAwB,EAAxB,mCAAwB;CAC3B,CAAC;AAWE,gCAAU;AATd,4BAA4B;AAC5B,MAAM,+BAA+B,GAAG,+BAAoB,CAAC;AAUzD,0EAA+B"} 1 + {"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/index.ts"],"names":[],"mappings":";;;AAAA,4DAAoH;AAQpH,uDAA0E;AA2HtE,qGA3HI,+BAAoB,OA2HJ;AAFpB,2FAzH0B,qBAAU,OAyH1B;AAvHd,MAAM,oBAAoB,GAAG,CAAC,MAAc,EAAE,EAAE;IAC5C,oBAAoB;IACpB,4BAA4B;IAC5B,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC;AACzC,CAAC,CAAA;AAED,MAAM,iBAAiB,GAAG;IACtB,uBAAY;IACZ,wBAAa;IACb,yBAAc;CACjB,CAAA;AA2GG,8CAAiB;AAzGrB,MAAM,cAAc,GAAG,CAAC,IAAY,EAAE,IAAY,EAAE,OAAiC,EAAwB,EAAE;IAE3G,MAAM,EACF,UAAU,GAAG,+BAAoB,EACjC,UAAU,GAAG,iBAAiB,EAC9B,OAAO,GAAG,KAAK,GAClB,GAAG,OAAO,IAAI,EAAE,CAAC;IAElB,MAAM,MAAM,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,IAAI,CAAC,CAAC;IACjE,IAAI,MAAM,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,IAAI,CAAC,CAAC;IAE/D,MAAM,QAAQ,GAAG,MAAM,CAAC,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC;IAEjE,IAAI,OAAO,EAAE;QACT,8JAA8J;QAC9J,qGAAqG;QACrG,2BAA2B;QAC3B,MAAM,GAAG,IAAA,kBAAU,EAAC,MAAM,EAAE,MAAM,CAAC,CAAC;KACvC;IAED,MAAM,YAAY,GAA0C,EAAE,CAAC;IAE/D,KAAK,MAAM,KAAK,IAAI,UAAU,EAAE;QAC5B,IAAI,KAAK,CAAC,OAAO,KAAK,SAAS,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE;YAC/D,SAAS;SACZ;QACD,MAAM,GAAG,GAAG,KAAK,CAAC,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QAC3C,MAAM,MAAM,GAAG,OAAO,GAAG,KAAK,QAAQ,CAAC,CAAC,CAAC,EAAC,KAAK,EAAE,GAAG,EAAC,CAAC,CAAC,CAAC,GAAG,CAAC;QAC5D,YAAY,CAAC,IAAI,CAAC;YACd,GAAG,MAAM;YACT,IAAI,EAAE,KAAK,CAAC,IAAI;SACnB,CAAC,CAAC;KACN;IAED,mCAAmC;IACnC,MAAM,WAAW,GAAG,oBAAoB,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;IAE1D,qBAAqB;IACrB,MAAM,SAAS,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,GAAG,GAAG,IAAI,CAAC,KAAK,EAAE,CAAC,CAAC,GAAG,YAAY,CAAC,MAAM,CAAC;IAChG,kCAAkC;IAClC,MAAM,iBAAiB,GAAG,SAAS,GAAG,IAAI,CAAC,GAAG,CAAC,WAAW,EAAE,EAAE,CAAC,CAAC;IAChE,MAAM,QAAQ,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC,GAAgD,EAAE,IAAI,EAAE,EAAE;QAC5F,MAAM,EAAC,IAAI,EAAE,KAAK,EAAE,GAAG,IAAI,EAAC,GAAG,IAAI,CAAC;QACpC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG;YACb,GAAG,IAAI;YACP,KAAK;SACR,CAAC;QACF,OAAO,GAAG,CAAC;IACf,CAAC,EAAE,EAAE,CAAC,CAAC;IACP,OAAO;QACH,UAAU,EAAE,QAAQ;QACpB,SAAS;QACT,iBAAiB;KACpB,CAAA;AACL,CAAC,CAAA;AAiDG,wCAAc;AA/CX,MAAM,UAAU,GAAG,CAAC,MAAc,EAAE,MAAc,EAAE,OAAiC,EAAU,EAAE;IACpG,wJAAwJ;IACxJ,kFAAkF;IAClF,MAAM,OAAO,GAAG,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAClC,MAAM,OAAO,GAAG,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAClC,MAAM,sBAAsB,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,GAA+C,EAAE,IAAI,EAAE,EAAE;QACpG,IAAI,SAAS,GAAG,CAAC,CAAC;QAClB,IAAI,SAAS,GAAuB,CAAC,CAAC;QACtC,IAAI,KAAK,GAAG,CAAC,CAAC;QACd,KAAK,MAAM,KAAK,IAAI,GAAG,CAAC,SAAS,EAAE;YAC/B,MAAM,MAAM,GAAG,cAAc,CAAC,IAAI,EAAE,KAAK,EAAE,EAAC,GAAG,OAAO,EAAE,OAAO,EAAE,KAAK,EAAC,CAAC,CAAC;YACzE,IAAI,MAAM,CAAC,SAAS,GAAG,SAAS,EAAE;gBAC9B,SAAS,GAAG,MAAM,CAAC,SAAS,CAAC;gBAC7B,SAAS,GAAG,KAAK,CAAC;aACrB;YACD,KAAK,EAAE,CAAC;SACX;QAED,MAAM,gBAAgB,GAAG,CAAC,GAAG,GAAG,CAAC,SAAS,CAAC,CAAC;QAC5C,IAAG,SAAS,IAAI,gBAAgB,CAAC,MAAM,GAAG,CAAC,EAAE;YACzC,gBAAgB,CAAC,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;SACzC;QACD,MAAM,OAAO,GAAG,SAAS,IAAI,GAAG,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC;QAEnH,OAAO,EAAC,OAAO,EAAE,OAAO,EAAE,SAAS,EAAE,gBAAgB,EAAC,CAAC;IAC3D,CAAC,EAAE,EAAC,OAAO,EAAE,EAAE,EAAE,SAAS,EAAE,OAAO,EAAC,CAAC,CAAC;IACtC,MAAM,yBAAyB,GAAG,sBAAsB,CAAC,OAAO,CAAC,MAAM,CAAC,sBAAsB,CAAC,SAAS,CAAC,CAAC;IAC1G,OAAO,yBAAyB,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AAC/C,CAAC,CAAA;AA5BY,QAAA,UAAU,cA4BtB;AAED,MAAM,oBAAoB,GAAG,CAAC,QAAiC,EAAE,EAAE;IAC/D,OAAO,CAAC,IAAY,EAAE,IAAY,EAAE,UAAmC,EAAE,EAAE,EAAE,CAAC,cAAc,CAAC,IAAI,EAAE,IAAI,EAAE,EAAC,GAAG,QAAQ,EAAE,GAAG,OAAO,EAAC,CAAC,CAAC;AACxI,CAAC,CAAA;AAgBG,oDAAoB;AAdxB,MAAM,UAAU,GAAG;IACf,YAAY,EAAZ,uBAAY;IACZ,aAAa,EAAb,wBAAa;IACb,cAAc,EAAd,yBAAc;IACd,wBAAwB,EAAxB,mCAAwB;CAC3B,CAAC;AAWE,gCAAU;AATd,4BAA4B;AAC5B,MAAM,+BAA+B,GAAG,+BAAoB,CAAC;AAUzD,0EAA+B"}
+2 -2
dist/commonjs/normalization/index.js
··· 3 3 exports.strDefaultTransforms = exports.transforms = exports.replaceMultiWhitespace = exports.removeWhitespace = exports.removePunctuation = exports.replaceUnicode = exports.trim = exports.lowercase = void 0; 4 4 const PUNCTUATION_REGEX = new RegExp(/[^\w\s]|_/g); 5 5 const WHITESPACE_REGEX = new RegExp(/\s/g); 6 - const MULTI_WHITESPACE_REGEX = new RegExp(/\s{2,}|\n/g); 6 + const MULTI_WHITESPACE_REGEX = new RegExp(/\s{2,}/g); 7 7 const lowercase = (str) => str.toLocaleLowerCase(); 8 8 exports.lowercase = lowercase; 9 9 const trim = (str) => str.trim(); ··· 14 14 exports.removePunctuation = removePunctuation; 15 15 const removeWhitespace = (str) => str.replace(WHITESPACE_REGEX, ''); 16 16 exports.removeWhitespace = removeWhitespace; 17 - const replaceMultiWhitespace = (str) => str.replace(MULTI_WHITESPACE_REGEX, ''); 17 + const replaceMultiWhitespace = (str) => str.replace(MULTI_WHITESPACE_REGEX, ' '); 18 18 exports.replaceMultiWhitespace = replaceMultiWhitespace; 19 19 const transforms = { 20 20 lowercase,
+1 -1
dist/commonjs/normalization/index.js.map
··· 1 - {"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/normalization/index.ts"],"names":[],"mappings":";;;AAEA,MAAM,iBAAiB,GAAG,IAAI,MAAM,CAAC,YAAY,CAAC,CAAC;AACnD,MAAM,gBAAgB,GAAG,IAAI,MAAM,CAAC,KAAK,CAAC,CAAC;AAC3C,MAAM,sBAAsB,GAAG,IAAI,MAAM,CAAC,YAAY,CAAC,CAAC;AAExD,MAAM,SAAS,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,iBAAiB,EAAE,CAAC;AAyB5E,8BAAS;AAxBb,MAAM,IAAI,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC;AAyB1D,oBAAI;AAxBR,MAAM,cAAc,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,OAAO,CAAC,kBAAkB,EAAE,EAAE,CAAC,CAAC;AAyB9G,wCAAc;AAxBlB,MAAM,iBAAiB,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,OAAO,CAAC,iBAAiB,EAAE,EAAE,CAAC,CAAC;AAyB/F,8CAAiB;AAxBrB,MAAM,gBAAgB,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,OAAO,CAAC,gBAAgB,EAAE,EAAE,CAAC,CAAC;AAyB7F,4CAAgB;AAxBpB,MAAM,sBAAsB,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,OAAO,CAAC,sBAAsB,EAAE,EAAE,CAAC,CAAC;AAyBzG,wDAAsB;AAvB1B,MAAM,UAAU,GAAG;IACf,SAAS;IACT,IAAI;IACJ,sBAAsB;IACtB,cAAc;IACd,gBAAgB;IAChB,iBAAiB;CACpB,CAAA;AAiBG,gCAAU;AAfd,MAAM,oBAAoB,GAA0B;IAChD,cAAc;IACd,iBAAiB;IACjB,IAAI;IACJ,sBAAsB;IACtB,SAAS;CACZ,CAAC;AAUE,oDAAoB"} 1 + {"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/normalization/index.ts"],"names":[],"mappings":";;;AAEA,MAAM,iBAAiB,GAAG,IAAI,MAAM,CAAC,YAAY,CAAC,CAAC;AACnD,MAAM,gBAAgB,GAAG,IAAI,MAAM,CAAC,KAAK,CAAC,CAAC;AAC3C,MAAM,sBAAsB,GAAG,IAAI,MAAM,CAAC,SAAS,CAAC,CAAC;AAErD,MAAM,SAAS,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,iBAAiB,EAAE,CAAC;AAyB5E,8BAAS;AAxBb,MAAM,IAAI,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC;AAyB1D,oBAAI;AAxBR,MAAM,cAAc,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,OAAO,CAAC,kBAAkB,EAAE,EAAE,CAAC,CAAC;AAyB9G,wCAAc;AAxBlB,MAAM,iBAAiB,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,OAAO,CAAC,iBAAiB,EAAE,EAAE,CAAC,CAAC;AAyB/F,8CAAiB;AAxBrB,MAAM,gBAAgB,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,OAAO,CAAC,gBAAgB,EAAE,EAAE,CAAC,CAAC;AAyB7F,4CAAgB;AAxBpB,MAAM,sBAAsB,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,OAAO,CAAC,sBAAsB,EAAE,GAAG,CAAC,CAAC;AAyB1G,wDAAsB;AAvB1B,MAAM,UAAU,GAAG;IACf,SAAS;IACT,IAAI;IACJ,sBAAsB;IACtB,cAAc;IACd,gBAAgB;IAChB,iBAAiB;CACpB,CAAA;AAiBG,gCAAU;AAfd,MAAM,oBAAoB,GAA0B;IAChD,cAAc;IACd,iBAAiB;IACjB,IAAI;IACJ,sBAAsB;IACtB,SAAS;CACZ,CAAC;AAUE,oDAAoB"}
+12
dist/esm/atomic.d.ts
··· 1 1 export interface StringComparisonOptions { 2 + /** 3 + * An array of transformations to apply to each string before comparing similarity 4 + * */ 2 5 transforms?: StringTransformFunc[]; 6 + /** 7 + * An array of strategies used to score similarity. All strategies scores are combined for an average high score. 8 + * */ 3 9 strategies?: ComparisonStrategy<ComparisonStrategyResultValue>[]; 10 + /** 11 + * Reorder second string so its token match order of first string as closely as possible 12 + * 13 + * Useful when only the differences in content are important, but not the order of the content 14 + * */ 15 + reorder?: boolean; 4 16 } 5 17 export interface StringSamenessResult { 6 18 strategies: {
+1
dist/esm/index.d.ts
··· 2 2 import { strDefaultTransforms, transforms } from "./normalization/index.js"; 3 3 declare const defaultStrategies: import("./atomic.js").ComparisonStrategy<import("./atomic.js").ComparisonStrategyResultObject>[]; 4 4 declare const stringSameness: (valA: string, valB: string, options?: StringComparisonOptions) => StringSamenessResult; 5 + export declare const reorderStr: (cleanA: string, cleanB: string, options?: StringComparisonOptions) => string; 5 6 declare const createStringSameness: (defaults: StringComparisonOptions) => (valA: string, valB: string, options?: StringComparisonOptions) => StringSamenessResult; 6 7 declare const strategies: { 7 8 diceStrategy: import("./atomic.js").ComparisonStrategy<import("./atomic.js").ComparisonStrategyResultObject>;
+35 -2
dist/esm/index.js
··· 11 11 cosineStrategy 12 12 ]; 13 13 const stringSameness = (valA, valB, options) => { 14 - const { transforms = strDefaultTransforms, strategies = defaultStrategies, } = options || {}; 14 + const { transforms = strDefaultTransforms, strategies = defaultStrategies, reorder = false, } = options || {}; 15 15 const cleanA = transforms.reduce((acc, curr) => curr(acc), valA); 16 - const cleanB = transforms.reduce((acc, curr) => curr(acc), valB); 16 + let cleanB = transforms.reduce((acc, curr) => curr(acc), valB); 17 17 const shortest = cleanA.length > cleanB.length ? cleanB : cleanA; 18 + if (reorder) { 19 + // we want to ignore order of tokens as much as possible (user does not care about differences in word order, just absolute differences in characters overall) 20 + // so we will reorder cleanB so its tokens match the order or tokens in cleanA as closely as possible 21 + // before we run strategies 22 + cleanB = reorderStr(cleanA, cleanB); 23 + } 18 24 const stratResults = []; 19 25 for (const strat of strategies) { 20 26 if (strat.isValid !== undefined && !strat.isValid(cleanA, cleanB)) { ··· 46 52 highScore, 47 53 highScoreWeighted, 48 54 }; 55 + }; 56 + export const reorderStr = (cleanA, cleanB, options) => { 57 + // to do the reordering we will use stringSameness with the provided strats to match against each token in cleanA and choose the closest token in cleanB 58 + // and add the end concat any remaining tokens from cleanB to the reordered string 59 + const aTokens = cleanA.split(' '); 60 + const bTokens = cleanB.split(' '); 61 + const orderedCandidateTokens = aTokens.reduce((acc, curr) => { 62 + let highScore = 0; 63 + let highIndex = 0; 64 + let index = 0; 65 + for (const token of acc.remaining) { 66 + const result = stringSameness(curr, token, { ...options, reorder: false }); 67 + if (result.highScore > highScore) { 68 + highScore = result.highScore; 69 + highIndex = index; 70 + } 71 + index++; 72 + } 73 + const splicedRemaining = [...acc.remaining]; 74 + if (highIndex <= splicedRemaining.length - 1) { 75 + splicedRemaining.splice(highIndex, 1); 76 + } 77 + const ordered = highIndex <= acc.remaining.length - 1 ? acc.ordered.concat(acc.remaining[highIndex]) : acc.ordered; 78 + return { ordered: ordered, remaining: splicedRemaining }; 79 + }, { ordered: [], remaining: bTokens }); 80 + const allOrderedCandidateTokens = orderedCandidateTokens.ordered.concat(orderedCandidateTokens.remaining); 81 + return allOrderedCandidateTokens.join(' '); 49 82 }; 50 83 const createStringSameness = (defaults) => { 51 84 return (valA, valB, options = {}) => stringSameness(valA, valB, { ...defaults, ...options });
+1 -1
dist/esm/index.js.map
··· 1 - {"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAC,cAAc,EAAE,aAAa,EAAE,YAAY,EAAE,wBAAwB,EAAC,MAAM,+BAA+B,CAAC;AAQpH,OAAO,EAAC,oBAAoB,EAAE,UAAU,EAAC,MAAM,0BAA0B,CAAC;AAE1E,MAAM,oBAAoB,GAAG,CAAC,MAAc,EAAE,EAAE;IAC5C,oBAAoB;IACpB,4BAA4B;IAC5B,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC;AACzC,CAAC,CAAA;AAED,MAAM,iBAAiB,GAAG;IACtB,YAAY;IACZ,aAAa;IACb,cAAc;CACjB,CAAA;AAED,MAAM,cAAc,GAAG,CAAC,IAAY,EAAE,IAAY,EAAE,OAAiC,EAAwB,EAAE;IAE3G,MAAM,EACF,UAAU,GAAG,oBAAoB,EACjC,UAAU,GAAG,iBAAiB,GACjC,GAAG,OAAO,IAAI,EAAE,CAAC;IAElB,MAAM,MAAM,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,IAAI,CAAC,CAAC;IACjE,MAAM,MAAM,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,IAAI,CAAC,CAAC;IAEjE,MAAM,QAAQ,GAAG,MAAM,CAAC,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC;IAEjE,MAAM,YAAY,GAA0C,EAAE,CAAC;IAE/D,KAAK,MAAM,KAAK,IAAI,UAAU,EAAE;QAC5B,IAAI,KAAK,CAAC,OAAO,KAAK,SAAS,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE;YAC/D,SAAS;SACZ;QACD,MAAM,GAAG,GAAG,KAAK,CAAC,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QAC3C,MAAM,MAAM,GAAG,OAAO,GAAG,KAAK,QAAQ,CAAC,CAAC,CAAC,EAAC,KAAK,EAAE,GAAG,EAAC,CAAC,CAAC,CAAC,GAAG,CAAC;QAC5D,YAAY,CAAC,IAAI,CAAC;YACd,GAAG,MAAM;YACT,IAAI,EAAE,KAAK,CAAC,IAAI;SACnB,CAAC,CAAC;KACN;IAED,mCAAmC;IACnC,MAAM,WAAW,GAAG,oBAAoB,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;IAE1D,qBAAqB;IACrB,MAAM,SAAS,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,GAAG,GAAG,IAAI,CAAC,KAAK,EAAE,CAAC,CAAC,GAAG,YAAY,CAAC,MAAM,CAAC;IAChG,kCAAkC;IAClC,MAAM,iBAAiB,GAAG,SAAS,GAAG,IAAI,CAAC,GAAG,CAAC,WAAW,EAAE,EAAE,CAAC,CAAC;IAChE,MAAM,QAAQ,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC,GAAgD,EAAE,IAAI,EAAE,EAAE;QAC5F,MAAM,EAAC,IAAI,EAAE,KAAK,EAAE,GAAG,IAAI,EAAC,GAAG,IAAI,CAAC;QACpC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG;YACb,GAAG,IAAI;YACP,KAAK;SACR,CAAC;QACF,OAAO,GAAG,CAAC;IACf,CAAC,EAAE,EAAE,CAAC,CAAC;IACP,OAAO;QACH,UAAU,EAAE,QAAQ;QACpB,SAAS;QACT,iBAAiB;KACpB,CAAA;AACL,CAAC,CAAA;AAED,MAAM,oBAAoB,GAAG,CAAC,QAAiC,EAAE,EAAE;IAC/D,OAAO,CAAC,IAAY,EAAE,IAAY,EAAE,UAAmC,EAAE,EAAE,EAAE,CAAC,cAAc,CAAC,IAAI,EAAE,IAAI,EAAE,EAAC,GAAG,QAAQ,EAAE,GAAG,OAAO,EAAC,CAAC,CAAC;AACxI,CAAC,CAAA;AAED,MAAM,UAAU,GAAG;IACf,YAAY;IACZ,aAAa;IACb,cAAc;IACd,wBAAwB;CAC3B,CAAC;AAEF,4BAA4B;AAC5B,MAAM,+BAA+B,GAAG,oBAAoB,CAAC;AAE7D,OAAO,EAGH,cAAc,EACd,oBAAoB,EACpB,iBAAiB,EACjB,UAAU,EACV,UAAU,EACV,+BAA+B,EAC/B,oBAAoB,EAGvB,CAAA"} 1 + {"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAC,cAAc,EAAE,aAAa,EAAE,YAAY,EAAE,wBAAwB,EAAC,MAAM,+BAA+B,CAAC;AAQpH,OAAO,EAAC,oBAAoB,EAAE,UAAU,EAAC,MAAM,0BAA0B,CAAC;AAE1E,MAAM,oBAAoB,GAAG,CAAC,MAAc,EAAE,EAAE;IAC5C,oBAAoB;IACpB,4BAA4B;IAC5B,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC;AACzC,CAAC,CAAA;AAED,MAAM,iBAAiB,GAAG;IACtB,YAAY;IACZ,aAAa;IACb,cAAc;CACjB,CAAA;AAED,MAAM,cAAc,GAAG,CAAC,IAAY,EAAE,IAAY,EAAE,OAAiC,EAAwB,EAAE;IAE3G,MAAM,EACF,UAAU,GAAG,oBAAoB,EACjC,UAAU,GAAG,iBAAiB,EAC9B,OAAO,GAAG,KAAK,GAClB,GAAG,OAAO,IAAI,EAAE,CAAC;IAElB,MAAM,MAAM,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,IAAI,CAAC,CAAC;IACjE,IAAI,MAAM,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,IAAI,CAAC,CAAC;IAE/D,MAAM,QAAQ,GAAG,MAAM,CAAC,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC;IAEjE,IAAI,OAAO,EAAE;QACT,8JAA8J;QAC9J,qGAAqG;QACrG,2BAA2B;QAC3B,MAAM,GAAG,UAAU,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;KACvC;IAED,MAAM,YAAY,GAA0C,EAAE,CAAC;IAE/D,KAAK,MAAM,KAAK,IAAI,UAAU,EAAE;QAC5B,IAAI,KAAK,CAAC,OAAO,KAAK,SAAS,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE;YAC/D,SAAS;SACZ;QACD,MAAM,GAAG,GAAG,KAAK,CAAC,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QAC3C,MAAM,MAAM,GAAG,OAAO,GAAG,KAAK,QAAQ,CAAC,CAAC,CAAC,EAAC,KAAK,EAAE,GAAG,EAAC,CAAC,CAAC,CAAC,GAAG,CAAC;QAC5D,YAAY,CAAC,IAAI,CAAC;YACd,GAAG,MAAM;YACT,IAAI,EAAE,KAAK,CAAC,IAAI;SACnB,CAAC,CAAC;KACN;IAED,mCAAmC;IACnC,MAAM,WAAW,GAAG,oBAAoB,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;IAE1D,qBAAqB;IACrB,MAAM,SAAS,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,GAAG,GAAG,IAAI,CAAC,KAAK,EAAE,CAAC,CAAC,GAAG,YAAY,CAAC,MAAM,CAAC;IAChG,kCAAkC;IAClC,MAAM,iBAAiB,GAAG,SAAS,GAAG,IAAI,CAAC,GAAG,CAAC,WAAW,EAAE,EAAE,CAAC,CAAC;IAChE,MAAM,QAAQ,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC,GAAgD,EAAE,IAAI,EAAE,EAAE;QAC5F,MAAM,EAAC,IAAI,EAAE,KAAK,EAAE,GAAG,IAAI,EAAC,GAAG,IAAI,CAAC;QACpC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG;YACb,GAAG,IAAI;YACP,KAAK;SACR,CAAC;QACF,OAAO,GAAG,CAAC;IACf,CAAC,EAAE,EAAE,CAAC,CAAC;IACP,OAAO;QACH,UAAU,EAAE,QAAQ;QACpB,SAAS;QACT,iBAAiB;KACpB,CAAA;AACL,CAAC,CAAA;AAED,MAAM,CAAC,MAAM,UAAU,GAAG,CAAC,MAAc,EAAE,MAAc,EAAE,OAAiC,EAAU,EAAE;IACpG,wJAAwJ;IACxJ,kFAAkF;IAClF,MAAM,OAAO,GAAG,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAClC,MAAM,OAAO,GAAG,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAClC,MAAM,sBAAsB,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,GAA+C,EAAE,IAAI,EAAE,EAAE;QACpG,IAAI,SAAS,GAAG,CAAC,CAAC;QAClB,IAAI,SAAS,GAAuB,CAAC,CAAC;QACtC,IAAI,KAAK,GAAG,CAAC,CAAC;QACd,KAAK,MAAM,KAAK,IAAI,GAAG,CAAC,SAAS,EAAE;YAC/B,MAAM,MAAM,GAAG,cAAc,CAAC,IAAI,EAAE,KAAK,EAAE,EAAC,GAAG,OAAO,EAAE,OAAO,EAAE,KAAK,EAAC,CAAC,CAAC;YACzE,IAAI,MAAM,CAAC,SAAS,GAAG,SAAS,EAAE;gBAC9B,SAAS,GAAG,MAAM,CAAC,SAAS,CAAC;gBAC7B,SAAS,GAAG,KAAK,CAAC;aACrB;YACD,KAAK,EAAE,CAAC;SACX;QAED,MAAM,gBAAgB,GAAG,CAAC,GAAG,GAAG,CAAC,SAAS,CAAC,CAAC;QAC5C,IAAG,SAAS,IAAI,gBAAgB,CAAC,MAAM,GAAG,CAAC,EAAE;YACzC,gBAAgB,CAAC,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;SACzC;QACD,MAAM,OAAO,GAAG,SAAS,IAAI,GAAG,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC;QAEnH,OAAO,EAAC,OAAO,EAAE,OAAO,EAAE,SAAS,EAAE,gBAAgB,EAAC,CAAC;IAC3D,CAAC,EAAE,EAAC,OAAO,EAAE,EAAE,EAAE,SAAS,EAAE,OAAO,EAAC,CAAC,CAAC;IACtC,MAAM,yBAAyB,GAAG,sBAAsB,CAAC,OAAO,CAAC,MAAM,CAAC,sBAAsB,CAAC,SAAS,CAAC,CAAC;IAC1G,OAAO,yBAAyB,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AAC/C,CAAC,CAAA;AAED,MAAM,oBAAoB,GAAG,CAAC,QAAiC,EAAE,EAAE;IAC/D,OAAO,CAAC,IAAY,EAAE,IAAY,EAAE,UAAmC,EAAE,EAAE,EAAE,CAAC,cAAc,CAAC,IAAI,EAAE,IAAI,EAAE,EAAC,GAAG,QAAQ,EAAE,GAAG,OAAO,EAAC,CAAC,CAAC;AACxI,CAAC,CAAA;AAED,MAAM,UAAU,GAAG;IACf,YAAY;IACZ,aAAa;IACb,cAAc;IACd,wBAAwB;CAC3B,CAAC;AAEF,4BAA4B;AAC5B,MAAM,+BAA+B,GAAG,oBAAoB,CAAC;AAE7D,OAAO,EAGH,cAAc,EACd,oBAAoB,EACpB,iBAAiB,EACjB,UAAU,EACV,UAAU,EACV,+BAA+B,EAC/B,oBAAoB,EAGvB,CAAA"}
+2 -2
dist/esm/normalization/index.js
··· 1 1 const PUNCTUATION_REGEX = new RegExp(/[^\w\s]|_/g); 2 2 const WHITESPACE_REGEX = new RegExp(/\s/g); 3 - const MULTI_WHITESPACE_REGEX = new RegExp(/\s{2,}|\n/g); 3 + const MULTI_WHITESPACE_REGEX = new RegExp(/\s{2,}/g); 4 4 const lowercase = (str) => str.toLocaleLowerCase(); 5 5 const trim = (str) => str.trim(); 6 6 const replaceUnicode = (str) => str.normalize('NFD').replace(/[\u0300-\u036f]/g, ""); 7 7 const removePunctuation = (str) => str.replace(PUNCTUATION_REGEX, ''); 8 8 const removeWhitespace = (str) => str.replace(WHITESPACE_REGEX, ''); 9 - const replaceMultiWhitespace = (str) => str.replace(MULTI_WHITESPACE_REGEX, ''); 9 + const replaceMultiWhitespace = (str) => str.replace(MULTI_WHITESPACE_REGEX, ' '); 10 10 const transforms = { 11 11 lowercase, 12 12 trim,
+1 -1
dist/esm/normalization/index.js.map
··· 1 - {"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/normalization/index.ts"],"names":[],"mappings":"AAEA,MAAM,iBAAiB,GAAG,IAAI,MAAM,CAAC,YAAY,CAAC,CAAC;AACnD,MAAM,gBAAgB,GAAG,IAAI,MAAM,CAAC,KAAK,CAAC,CAAC;AAC3C,MAAM,sBAAsB,GAAG,IAAI,MAAM,CAAC,YAAY,CAAC,CAAC;AAExD,MAAM,SAAS,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,iBAAiB,EAAE,CAAC;AAChF,MAAM,IAAI,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC;AAC9D,MAAM,cAAc,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,OAAO,CAAC,kBAAkB,EAAE,EAAE,CAAC,CAAC;AAClH,MAAM,iBAAiB,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,OAAO,CAAC,iBAAiB,EAAE,EAAE,CAAC,CAAC;AACnG,MAAM,gBAAgB,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,OAAO,CAAC,gBAAgB,EAAE,EAAE,CAAC,CAAC;AACjG,MAAM,sBAAsB,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,OAAO,CAAC,sBAAsB,EAAE,EAAE,CAAC,CAAC;AAE7G,MAAM,UAAU,GAAG;IACf,SAAS;IACT,IAAI;IACJ,sBAAsB;IACtB,cAAc;IACd,gBAAgB;IAChB,iBAAiB;CACpB,CAAA;AAED,MAAM,oBAAoB,GAA0B;IAChD,cAAc;IACd,iBAAiB;IACjB,IAAI;IACJ,sBAAsB;IACtB,SAAS;CACZ,CAAC;AAEF,OAAO,EACH,SAAS,EACT,IAAI,EACJ,cAAc,EACd,iBAAiB,EACjB,gBAAgB,EAChB,sBAAsB,EACtB,UAAU,EACV,oBAAoB,EACvB,CAAA"} 1 + {"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/normalization/index.ts"],"names":[],"mappings":"AAEA,MAAM,iBAAiB,GAAG,IAAI,MAAM,CAAC,YAAY,CAAC,CAAC;AACnD,MAAM,gBAAgB,GAAG,IAAI,MAAM,CAAC,KAAK,CAAC,CAAC;AAC3C,MAAM,sBAAsB,GAAG,IAAI,MAAM,CAAC,SAAS,CAAC,CAAC;AAErD,MAAM,SAAS,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,iBAAiB,EAAE,CAAC;AAChF,MAAM,IAAI,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC;AAC9D,MAAM,cAAc,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,OAAO,CAAC,kBAAkB,EAAE,EAAE,CAAC,CAAC;AAClH,MAAM,iBAAiB,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,OAAO,CAAC,iBAAiB,EAAE,EAAE,CAAC,CAAC;AACnG,MAAM,gBAAgB,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,OAAO,CAAC,gBAAgB,EAAE,EAAE,CAAC,CAAC;AACjG,MAAM,sBAAsB,GAAwB,CAAC,GAAW,EAAE,EAAE,CAAC,GAAG,CAAC,OAAO,CAAC,sBAAsB,EAAE,GAAG,CAAC,CAAC;AAE9G,MAAM,UAAU,GAAG;IACf,SAAS;IACT,IAAI;IACJ,sBAAsB;IACtB,cAAc;IACd,gBAAgB;IAChB,iBAAiB;CACpB,CAAA;AAED,MAAM,oBAAoB,GAA0B;IAChD,cAAc;IACd,iBAAiB;IACjB,IAAI;IACJ,sBAAsB;IACtB,SAAS;CACZ,CAAC;AAEF,OAAO,EACH,SAAS,EACT,IAAI,EACJ,cAAc,EACd,iBAAiB,EACjB,gBAAgB,EAChB,sBAAsB,EACtB,UAAU,EACV,oBAAoB,EACvB,CAAA"}