userdiff: support Markdown
PATCH 1/1 userdiff: support Markdown
It's typical to find Markdown documentation alongside source code, and having better context for documentation changes is useful; see also commit 69f9c87d4 (userdiff: add support for Fountain documents, 2015-07-21).
The pattern is based on the CommonMark specification 0.29, section 4.2:
https://spec.commonmark.org/
Only ATX headings are supported, as detecting setext headings would require printing the line before a pattern matches, or matching a multiline pattern.
---
If it is indeed possible to match multiline patterns, let me know! I would love to support setext (underlined) headings with this.
I would also appreciate feedback on the word-diff pattern here, I have no real idea what should constitute a word in a Markdown document, apart from that it should probably be similar to the definition given for Fountain, given that Fountain appears to have somewhat similar inline syntax to Markdown.
Documentation/gitattributes.txt | 2 ++ t/t4018-diff-funcname.sh | 1 + t/t4018/markdown-heading-indented | 6 ++++++ t/t4018/markdown-heading-non-headings | 17 +++++++++++++++++ userdiff.c | 3 +++ 5 files changed, 29 insertions(+) create mode 100644 t/t4018/markdown-heading-indented create mode 100644 t/t4018/markdown-heading-non-headings
Documentation/gitattributes.txt +2 -0
@@ -824,6 +824,8 @@ patterns are available: - `java` suitable for source code in the Java language. +- `markdown` suitable for Markdown documents. + - `matlab` suitable for source code in the MATLAB and Octave languages. - `objc` suitable for source code in the Objective-C language.
t/t4018-diff-funcname.sh +1 -0
@@ -38,6 +38,7 @@ diffpatterns=" golang html java + markdown matlab objc pascal
t/t4018/markdown-heading-indented new file +6 -0
@@ -0,0 +1,6 @@ +Indented headings are allowed, as long as the indent is no more than 3 spaces. + + ### RIGHT + +- something +- ChangeMe
t/t4018/markdown-heading-non-headings new file +17 -0
@@ -0,0 +1,17 @@ +Headings can be right next to other lines of the file: +# RIGHT +Indents of more than four spaces make a code block: + + # code comment, not heading + +If there's no space after the final hash, it's not a heading: + +#hashtag + +Sequences of more than 6 hashes don't make a heading: + +####### over-enthusiastic heading + +So the detected heading should be right up at the start of this file. + +ChangeMe
userdiff.c +3 -0
@@ -79,6 +79,9 @@ PATTERNS("java","|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?" "|[-+*/<>%&^|=!]=" "|--|\\+\\+|<<=?|>>>?=?|&&|\\|\\|"), +PATTERNS ("markdown" , +"^ {0,3}#{1,6}( .*)?$" , +"[^ \t-]+" ), PATTERNS("matlab",/* * Octave pattern is mostly the same as matlab, except that '%%%' and
Signature · 1 line
2.26.1
Quoted 5 lines · Ash Holland
I would also appreciate feedback on the word-diff pattern here, I have no real idea what should constitute a word in a Markdown document, apart from that it should probably be similar to the definition given for Fountain, given that Fountain appears to have somewhat similar inline syntax to Markdown.
Since Markdown can have raw HTML tags in many variants, it may make sense to extend the word pattern to "[^<>= \t]+" like HTML's pattern so tags starting/ending will not be considered part of a word.
Quoted 3 lines · Emma Brooks
Since Markdown can have raw HTML tags in many variants, it may make sense to extend the word pattern to "[^<>= \t]+" like HTML's pattern so tags starting/ending will not be considered part of a word.
Good point, I'll update the pattern to that, thanks!
Quoted 6 lines · Ash Holland
Quoted 3 lines · Emma Brooks
Since Markdown can have raw HTML tags in many variants, it may make sense to extend the word pattern to "[^<>= \t]+" like HTML's pattern so tags starting/ending will not be considered part of a word.
Good point, I'll update the pattern to that, thanks!
I just marked the topic as "expecting a reroll" in the "What's cooking" report I have been preparing, but has something happened after this exchange?
No need to rush, but we'll be closing the acceptance of new features in three weeks for this cycle, so we won't have infinite amount of time, either.
Thanks.
Quoted 7 lines · Junio C Hamano
Quoted 5 lines · Ash Holland
Quoted 3 lines · Emma Brooks
Since Markdown can have raw HTML tags in many variants, it may make sense to extend the word pattern to "[^<>= \t]+" like HTML's pattern so tags starting/ending will not be considered part of a word.
Quoted 5 lines
Quoted 1 line
Good point, I'll update the pattern to that, thanks!
I just marked the topic as "expecting a reroll" in the "What's cooking" report I have been preparing, but has something happened after this exchange?
You've not missed anything, sorry, I've been busy with exams -- I'll prepare a v2 later today.
Am 21.04.20 um 03:00 schrieb Ash Holland:
Quoted 11 lines
It's typical to find Markdown documentation alongside source code, and having better context for documentation changes is useful; see also commit
69f9c87d4(userdiff: add support for Fountain documents, 2015-07-21).The pattern is based on the CommonMark specification 0.29, section 4.2:
https://spec.commonmark.org/Only ATX headings are supported, as detecting setext headings would require printing the line before a pattern matches, or matching a multiline pattern.
The patch looks good. I have one question about the patthern, though (see below).
Quoted 5 lines
Signed-off-by Ash Holland <[email protected]>---
If it is indeed possible to match multiline patterns, let me know! I would love to support setext (underlined) headings with this.
We don't have multi-line matching, unfortunately.
Quoted 65 lines
I would also appreciate feedback on the word-diff pattern here, I have no real idea what should constitute a word in a Markdown document, apart from that it should probably be similar to the definition given for Fountain, given that Fountain appears to have somewhat similar inline syntax to Markdown.
Documentation/gitattributes.txt | 2 ++ t/t4018-diff-funcname.sh | 1 + t/t4018/markdown-heading-indented | 6 ++++++ t/t4018/markdown-heading-non-headings | 17 +++++++++++++++++ userdiff.c | 3 +++ 5 files changed, 29 insertions(+) create mode 100644 t/t4018/markdown-heading-indented create mode 100644 t/t4018/markdown-heading-non-headingsDocumentation/gitattributes.txt +2
-0@@ -824,6 +824,8 @@ patterns are available: - `java` suitable for source code in the Java language. +- `markdown` suitable for Markdown documents. + - `matlab` suitable for source code in the MATLAB and Octave languages. - `objc` suitable for source code in the Objective-C language.t/t4018-diff-funcname.sh +1
-0@@ -38,6 +38,7 @@ diffpatterns=" golang html java + markdown matlab objc pascalt/t4018/markdown-heading-indented new file +6
-0@@ -0,0 +1,6 @@ +Indented headings are allowed, as long as the indent is no more than 3 spaces. + + ### RIGHT + +- something +- ChangeMet/t4018/markdown-heading-non-headings new file +7
-0@@ -0,0 +1,17 @@ +Headings can be right next to other lines of the file: +# RIGHT +Indents of more than four spaces make a code block: + + # code comment, not heading + +If there's no space after the final hash, it's not a heading:
t/t4018/markdown-heading-non-headings line 17
Quoted 10 lines
t/t4018/markdown-heading-non-headings new file +10
-0+ +#hashtag + +Sequences of more than 6 hashes don't make a heading: + +####### over-enthusiastic heading + +So the detected heading should be right up at the start of this file. + +ChangeMe
Nicely done!
userdiff.c PATTERNS("java", · line 83
Quoted 10 lines
userdiff.c +2
-0@@ -79,6 +79,9 @@ PATTERNS("java","|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?" "|[-+*/<>%&^|=!]=" "|--|\\+\\+|<<=?|>>>?=?|&&|\\|\\|"), +PATTERNS("markdown", + "^ {0,3}#{1,6}( .*)?$",
What is the purpose of making the heading text optional? Why would you want to match a sequence of hash marks without any text following it?
userdiff.c PATTERNS("java", · line 87
Quoted 5 lines
userdiff.c +1
-0+ "[^ \t-]+"), PATTERNS("matlab",/* * Octave pattern is mostly the same as matlab, except that '%%%' and
-- Hannes
Quoted 14 lines · Johannes Sixt
Am 21.04.20 um 03:00 schrieb Ash Holland:
userdiff.c PATTERNS("java", · line 83
Quoted 10 lines
userdiff.c +2
-0@@ -79,6 +79,9 @@ PATTERNS("java","|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?" "|[-+*/<>%&^|=!]=" "|--|\\+\\+|<<=?|>>>?=?|&&|\\|\\|"), +PATTERNS("markdown", + "^ {0,3}#{1,6}( .*)?$",What is the purpose of making the heading text optional? Why would you want to match a sequence of hash marks without any text following it?
Strictly speaking, a markdown heading is allowed to be empty -- see for example https://spec.commonmark.org/0.29/#example-49. I'm happy to change it if you think it's more useful to show a previous heading which contains text than an empty one, though.
Am 24.04.20 um 01:42 schrieb Ash Holland:
Quoted 16 lines · Johannes Sixt
Quoted 14 lines
Am 21.04.20 um 03:00 schrieb Ash Holland:
userdiff.c PATTERNS("java", · line 83
Quoted 10 lines
userdiff.c +2
-0@@ -79,6 +79,9 @@ PATTERNS("java","|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?" "|[-+*/<>%&^|=!]=" "|--|\\+\\+|<<=?|>>>?=?|&&|\\|\\|"), +PATTERNS("markdown", + "^ {0,3}#{1,6}( .*)?$",What is the purpose of making the heading text optional? Why would you want to match a sequence of hash marks without any text following it?
Quoted 4 lines
Strictly speaking, a markdown heading is allowed to be empty -- see for example https://spec.commonmark.org/0.29/#example-49. I'm happy to change it if you think it's more useful to show a previous heading which contains text than an empty one, though.
I don't know what makes sense, I don't write markdown regularly. A quick check shows that the sequence of hashmarks appears in the hunk header. Is that useful? (A genuine question!)
-- Hannes
Quoted 17 lines · Johannes Sixt
Am 24.04.20 um 01:42 schrieb Ash Holland:
Quoted 16 lines · Johannes Sixt
Quoted 14 lines
Am 21.04.20 um 03:00 schrieb Ash Holland:
userdiff.c PATTERNS("java", · line 83
Quoted 10 lines
userdiff.c +2
-0@@ -79,6 +79,9 @@ PATTERNS("java","|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?" "|[-+*/<>%&^|=!]=" "|--|\\+\\+|<<=?|>>>?=?|&&|\\|\\|"), +PATTERNS("markdown", + "^ {0,3}#{1,6}( .*)?$",What is the purpose of making the heading text optional? Why would you want to match a sequence of hash marks without any text following it?
Quoted 8 lines
Quoted 4 lines
Strictly speaking, a markdown heading is allowed to be empty -- see for example https://spec.commonmark.org/0.29/#example-49. I'm happy to change it if you think it's more useful to show a previous heading which contains text than an empty one, though.
I don't know what makes sense, I don't write markdown regularly. A quick check shows that the sequence of hashmarks appears in the hunk header. Is that useful? (A genuine question!)
I think probably it would be more confusing to have Git silently ignore empty headings, having occasionally written documents with empty headings in the past (e.g. when I know I want some different sections, but I don't know what to call them yet). Probably not many people would ever run into this situation either way, though.
PATCH v2 1/1 userdiff: support Markdown
In reply to Ash HollandIt's typical to find Markdown documentation alongside source code, and having better context for documentation changes is useful; see also commit 69f9c87d4 (userdiff: add support for Fountain documents, 2015-07-21).
The pattern is based on the CommonMark specification 0.29, section 4.2:
https://spec.commonmark.org/
Only ATX headings are supported, as detecting setext headings would require printing the line before a pattern matches, or matching a multiline pattern. The word-diff pattern is the same as the pattern for HTML, because many Markdown parsers accept inline HTML.
---
Changes since the previous patch:
- changed the word-diff pattern to match the HTML pattern
- fixed an off-by-one error in the wording of the test
Documentation/gitattributes.txt | 2 ++ t/t4018-diff-funcname.sh | 1 + t/t4018/markdown-heading-indented | 6 ++++++ t/t4018/markdown-heading-non-headings | 17 +++++++++++++++++ userdiff.c | 3 +++ 5 files changed, 29 insertions(+) create mode 100644 t/t4018/markdown-heading-indented create mode 100644 t/t4018/markdown-heading-non-headings
Documentation/gitattributes.txt +2 -0
@@ -824,6 +824,8 @@ patterns are available: - `java` suitable for source code in the Java language. +- `markdown` suitable for Markdown documents. + - `matlab` suitable for source code in the MATLAB and Octave languages. - `objc` suitable for source code in the Objective-C language.
t/t4018-diff-funcname.sh +1 -0
@@ -38,6 +38,7 @@ diffpatterns=" golang html java + markdown matlab objc pascal
t/t4018/markdown-heading-indented new file +6 -0
@@ -0,0 +1,6 @@ +Indented headings are allowed, as long as the indent is no more than 3 spaces. + + ### RIGHT + +- something +- ChangeMe
t/t4018/markdown-heading-non-headings new file +17 -0
@@ -0,0 +1,17 @@ +Headings can be right next to other lines of the file: +# RIGHT +Indents of four or more spaces make a code block: + + # code comment, not heading + +If there's no space after the final hash, it's not a heading: + +#hashtag + +Sequences of more than 6 hashes don't make a heading: + +####### over-enthusiastic heading + +So the detected heading should be right up at the start of this file. + +ChangeMe
userdiff.c +3 -0
@@ -79,6 +79,9 @@ PATTERNS("java","|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?" "|[-+*/<>%&^|=!]=" "|--|\\+\\+|<<=?|>>>?=?|&&|\\|\\|"), +PATTERNS ("markdown" , +"^ {0,3}#{1,6}( .*)?$" , +"[^<>= \t]+" ), PATTERNS("matlab",/* * Octave pattern is mostly the same as matlab, except that '%%%' and
Signature · 1 line
2.26.2
Quoted 13 lines · Ash Holland
It's typical to find Markdown documentation alongside source code, and having better context for documentation changes is useful; see also commit
69f9c87d4(userdiff: add support for Fountain documents, 2015-07-21).The pattern is based on the CommonMark specification 0.29, section 4.2:
https://spec.commonmark.org/Only ATX headings are supported, as detecting setext headings would require printing the line before a pattern matches, or matching a multiline pattern. The word-diff pattern is the same as the pattern for HTML, because many Markdown parsers accept inline HTML.
userdiff.c PATTERNS("java", · line 83
Quoted 2 lines
userdiff.c +2
-0+PATTERNS("markdown", + "^ {0,3}#{1,6}( .*)?$",
This is "possibly just a bit indented run of up to 6 hashes, either ending the line by itself or if some text follows, there must be a SP after the hashes".
If I had a line that has a hash, HT and then "Hello, world", would everybody's markdown implementation reject it as a header, because the whitespace after the run of hashes is not a SP?
Also, allowing only the hashes might be spec-compliant, but how useful would it be to see just a sequence of 4 hashes without any text after "@@ -100,5, +100,6 @@" in the diff output?
Taking all that together, my suspicion is
"^ {0,3}#{1,6}[ \t]"i.e. "possibly slightly indented run of 6 hashes, with a whitespace to catch the headers with real contents and nothing else" might be more practically useful. I dunno.
userdiff.c PATTERNS("java", · line 84
Quoted 1 line
userdiff.c +1
-0+ "[^<>= \t]+"),
This does match the one for HTML.
In any case, let me queue this v2 as-is and see what happens.
Thanks.
Quoted 7 lines · Junio C Hamano
This is "possibly just a bit indented run of up to 6 hashes, either ending the line by itself or if some text follows, there must be a SP after the hashes".
If I had a line that has a hash, HT and then "Hello, world", would everybody's markdown implementation reject it as a header, because the whitespace after the run of hashes is not a SP?
Good point, Pandoc at least accepts a tab as well as a space there. (Some markdown implementations don't even require the whitespace, so any line starting with #{1,6} would be a heading, but that runs into problems with paragraphs containing a line starting with "#", which is not uncommon.)
Quoted 11 lines
Also, allowing only the hashes might be spec-compliant, but how useful would it be to see just a sequence of 4 hashes without any text after "@@ -100,5, +100,6 @@" in the diff output?
Taking all that together, my suspicion is
"^ {0,3}#{1,6}[ \t]"
i.e. "possibly slightly indented run of 6 hashes, with a whitespace to catch the headers with real contents and nothing else" might be more practically useful. I dunno.
Sure, that looks plausible. I don't have a strong opinion on whether it's more useful to be consistent (and show the last heading, even if it doesn't contain any text) or to try as hard as possible to just show "some text", even if it's not the last heading, but two people have now suggested changing it, so I'll submit a v3 with your suggested pattern.
Am 01.05.20 um 13:49 schrieb Ash Holland:
Quoted 9 lines · Junio C Hamano
Quoted 7 lines
Taking all that together, my suspicion is
"^ {0,3}#{1,6}[ \t]"
i.e. "possibly slightly indented run of 6 hashes, with a whitespace to catch the headers with real contents and nothing else" might be more practically useful. I dunno.
Quoted 5 lines
Sure, that looks plausible. I don't have a strong opinion on whether it's more useful to be consistent (and show the last heading, even if it doesn't contain any text) or to try as hard as possible to just show "some text", even if it's not the last heading, but two people have now suggested changing it, so I'll submit a v3 with your suggested pattern.
The pattern above captures only the hashmarks, but not the text of the header. I suggest to append ".*".
-- Hannes
PATCH v3 1/1 userdiff: support Markdown
In reply to Ash HollandIt's typical to find Markdown documentation alongside source code, and having better context for documentation changes is useful; see also commit 69f9c87d4 (userdiff: add support for Fountain documents, 2015-07-21).
The pattern is based on the CommonMark specification 0.29, section 4.2
<https://spec.commonmark.org/> but doesn't match empty headings, as seeing them in a hunk header is unlikely to be useful.
Only ATX headings are supported, as detecting setext headings would require printing the line before a pattern matches, or matching a multiline pattern. The word-diff pattern is the same as the pattern for HTML, because many Markdown parsers accept inline HTML.
--- Documentation/gitattributes.txt | 2 ++ t/t4018-diff-funcname.sh | 1 + t/t4018/markdown-heading-indented | 6 ++++++ t/t4018/markdown-heading-non-headings | 17 +++++++++++++++++ userdiff.c | 3 +++ 5 files changed, 29 insertions(+) create mode 100644 t/t4018/markdown-heading-indented create mode 100644 t/t4018/markdown-heading-non-headings
Documentation/gitattributes.txt +2 -0
@@ -824,6 +824,8 @@ patterns are available: - `java` suitable for source code in the Java language. +- `markdown` suitable for Markdown documents. + - `matlab` suitable for source code in the MATLAB and Octave languages. - `objc` suitable for source code in the Objective-C language.
t/t4018-diff-funcname.sh +1 -0
@@ -38,6 +38,7 @@ diffpatterns=" golang html java + markdown matlab objc pascal
t/t4018/markdown-heading-indented new file +6 -0
@@ -0,0 +1,6 @@ +Indented headings are allowed, as long as the indent is no more than 3 spaces. + + ### RIGHT + +- something +- ChangeMe
t/t4018/markdown-heading-non-headings new file +17 -0
@@ -0,0 +1,17 @@ +Headings can be right next to other lines of the file: +# RIGHT +Indents of four or more spaces make a code block: + + # code comment, not heading + +If there's no space after the final hash, it's not a heading: + +#hashtag + +Sequences of more than 6 hashes don't make a heading: + +####### over-enthusiastic heading + +So the detected heading should be right up at the start of this file. + +ChangeMe
userdiff.c +3 -0
@@ -79,6 +79,9 @@ PATTERNS("java","|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?" "|[-+*/<>%&^|=!]=" "|--|\\+\\+|<<=?|>>>?=?|&&|\\|\\|"), +PATTERNS ("markdown" , +"^ {0,3}#{1,6}[ \t].*" , +"[^<>= \t]+" ), PATTERNS("matlab",/* * Octave pattern is mostly the same as matlab, except that '%%%' and
Signature · 1 line
2.26.2
Am 02.05.20 um 15:15 schrieb Ash Holland:
Quoted 91 lines
It's typical to find Markdown documentation alongside source code, and having better context for documentation changes is useful; see also commit
69f9c87d4(userdiff: add support for Fountain documents, 2015-07-21).The pattern is based on the CommonMark specification 0.29, section 4.2
<https://spec.commonmark.org/> but doesn't match empty headings, as seeing them in a hunk header is unlikely to be useful.Only ATX headings are supported, as detecting setext headings would require printing the line before a pattern matches, or matching a multiline pattern. The word-diff pattern is the same as the pattern for HTML, because many Markdown parsers accept inline HTML.
Signed-off-by Ash Holland <[email protected]>--- Documentation/gitattributes.txt | 2 ++ t/t4018-diff-funcname.sh | 1 + t/t4018/markdown-heading-indented | 6 ++++++ t/t4018/markdown-heading-non-headings | 17 +++++++++++++++++ userdiff.c | 3 +++ 5 files changed, 29 insertions(+) create mode 100644 t/t4018/markdown-heading-indented create mode 100644 t/t4018/markdown-heading-non-headingsDocumentation/gitattributes.txt +2
-0@@ -824,6 +824,8 @@ patterns are available: - `java` suitable for source code in the Java language. +- `markdown` suitable for Markdown documents. + - `matlab` suitable for source code in the MATLAB and Octave languages. - `objc` suitable for source code in the Objective-C language.t/t4018-diff-funcname.sh +1
-0@@ -38,6 +38,7 @@ diffpatterns=" golang html java + markdown matlab objc pascalt/t4018/markdown-heading-indented new file +6
-0@@ -0,0 +1,6 @@ +Indented headings are allowed, as long as the indent is no more than 3 spaces. + + ### RIGHT + +- something +- ChangeMet/t4018/markdown-heading-non-headings new file +17
-0@@ -0,0 +1,17 @@ +Headings can be right next to other lines of the file: +# RIGHT +Indents of four or more spaces make a code block: + + # code comment, not heading + +If there's no space after the final hash, it's not a heading: + +#hashtag + +Sequences of more than 6 hashes don't make a heading: + +####### over-enthusiastic heading + +So the detected heading should be right up at the start of this file. + +ChangeMeuserdiff.c +0
-0
userdiff.c PATTERNS("java", · line 87
Quoted 9 lines
userdiff.c +3
-0@@ -79,6 +79,9 @@ PATTERNS("java","|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?" "|[-+*/<>%&^|=!]=" "|--|\\+\\+|<<=?|>>>?=?|&&|\\|\\|"), +PATTERNS ("markdown" , +"^ {0,3}#{1,6}[ \t].*" , +"[^<>= \t]+" ), PATTERNS("matlab",/* * Octave pattern is mostly the same as matlab, except that '%%%' and
I tested this patch, and it looks good:
-- Hannes