feat: count string by codepoint #44
Conversation
yumetodo
force-pushed
the
feat/count_by_codepoint
branch
from
3ee7a33
to
ad37387
Compare
yumetodo
changed the title
yumetodo
force-pushed
the
feat/count_by_codepoint
branch
2 times, most recently
from
1ece1c7
to
ac71454
Compare
yumetodo
changed the title
yumetodo
force-pushed
the
feat/count_by_codepoint
branch
from
ac71454
to
c32cb9f
Compare
Unicode says that there are 4 ways to count string length. https://unicode.org/faq/char_combmark.html#7 This commit supports counting by Code points.
yumetodo
force-pushed
the
feat/count_by_codepoint
branch
from
c32cb9f
to
3bc67bf
Compare
There was a problem hiding this comment.
FYI: new Intl.Segmenter("ja-JP", { granularity: "grapheme" }) is more precise, but also more complex to implement due to language dependencies.
(probably, Intl.Segmenter is slower than others)
https://blog.jxck.io/entries/2017-03-02/unicode-in-javascript.html#unicode-text-segmentation
https://github.com/tc39/proposal-intl-segmenter
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/Segmenter
src/sentence-length.ts
Outdated
| * By default or set to "code units", count string by UTF-16 code unit(= using `String.prototype.length`). | ||
| * If set to "codepoints", count string by codepoint. | ||
| */ | ||
| countBy?: "code units" | "codepoints"; |
There was a problem hiding this comment.
| countBy?: "code units" | "codepoints"; | |
| countBy?: "codeunits" | "codepoints"; |
I think it would be better to align without spaces.
src/sentence-length.ts
Outdated
| const reporter: TextlintRuleReporter<Options> = (context, options = {}) => { | ||
| const maxLength = options.max ?? defaultOptions.max; | ||
| const skipPatterns = options.skipPatterns ?? options.exclusionPatterns ?? defaultOptions.skipPatterns; | ||
| const skipUrlStringLink = options.skipUrlStringLink ?? defaultOptions.skipUrlStringLink; | ||
| const strLen = | ||
| options.countBy == null || options.countBy === "code units" ? (s: string) => s.length : strLenByCodePoint; |
There was a problem hiding this comment.
Can you create a function like strLenByCodeUnits and use it?
const countBy = options?.countBy ?? defaultOptions.countBy;
const strLen = countBy === "codeunits" ? strLenByCodeUnits : strLenByCodePoint;
ref: - textlint-rule#44 (comment) Co-authored-by: azu <[email protected]>
ref: - textlint-rule#44 (comment) Co-authored-by: azu <[email protected]>
|
@azu Thank you for your review! I applied your suggestions.
I just now noticed the API. When we pass However, I think it's out of this PR's scope. |
Yes, I agree. |
Abstruct
Unicode says that there are 4 ways to count string length. https://unicode.org/faq/char_combmark.html#7
This commit supports counting by Code points.
Motivation
When we write text something like Japanese, surrogate pair will be used as usual. In such context, restricting string length is painful without considering surrogate pair.