Browse Source

Add KaTeX rendering to Markdown. (#20571)

This PR adds mathematical rendering with KaTeX.

The first step is to add a Goldmark extension that detects the latex
(and tex) mathematics delimiters.

The second step to make this extension only run if math support is
enabled.

The second step is to then add KaTeX CSS and JS to the head which will
load after the dom is rendered.

Fix #3445

Signed-off-by: Andrew Thornton <art27@cantab.net>

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
pull/21168/head
zeripath 3 weeks ago committed by GitHub
parent
commit
88c2e24360
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 3
      custom/conf/app.example.ini
  2. 1
      docs/content/doc/advanced/config-cheat-sheet.en-us.md
  3. 3
      docs/content/doc/advanced/external-renderers.en-us.md
  4. 2
      docs/content/doc/features/comparison.en-us.md
  5. 3
      docs/content/page/index.en-us.md
  6. 2
      go.mod
  7. 84
      modules/markup/markdown/convertyaml.go
  8. 15
      modules/markup/markdown/goldmark.go
  9. 20
      modules/markup/markdown/markdown.go
  10. 42
      modules/markup/markdown/math/block_node.go
  11. 123
      modules/markup/markdown/math/block_parser.go
  12. 43
      modules/markup/markdown/math/block_renderer.go
  13. 49
      modules/markup/markdown/math/inline_node.go
  14. 99
      modules/markup/markdown/math/inline_parser.go
  15. 47
      modules/markup/markdown/math/inline_renderer.go
  16. 108
      modules/markup/markdown/math/math.go
  17. 96
      modules/markup/markdown/meta.go
  18. 32
      modules/markup/markdown/meta_test.go
  19. 219
      modules/markup/markdown/renderconfig.go
  20. 162
      modules/markup/markdown/renderconfig_test.go
  21. 4
      modules/markup/sanitizer.go
  22. 2
      modules/setting/setting.go
  23. 39
      package-lock.json
  24. 1
      package.json
  25. 2
      web_src/js/markup/content.js
  26. 37
      web_src/js/markup/math.js
  27. 7
      web_src/less/animations.less
  28. 4
      webpack.config.js

3
custom/conf/app.example.ini

@ -1262,6 +1262,9 @@ ROUTER = console
;; List of file extensions that should be rendered/edited as Markdown
;; Separate the extensions with a comma. To render files without any extension as markdown, just put a comma
;FILE_EXTENSIONS = .md,.markdown,.mdown,.mkd
;;
;; Enables math inline and block detection
;ENABLE_MATH = true
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

1
docs/content/doc/advanced/config-cheat-sheet.en-us.md

@ -236,6 +236,7 @@ The following configuration set `Content-Type: application/vnd.android.package-a
- `CUSTOM_URL_SCHEMES`: Use a comma separated list (ftp,git,svn) to indicate additional
URL hyperlinks to be rendered in Markdown. URLs beginning in http and https are
always displayed
- `ENABLE_MATH`: **true**: Enables detection of `\(...\)`, `\[...\]`, `$...$` and `$$...$$` blocks as math blocks.
## Server (`server`)

3
docs/content/doc/advanced/external-renderers.en-us.md

@ -74,12 +74,13 @@ RENDER_COMMAND = "timeout 30s pandoc +RTS -M512M -RTS -f rst"
IS_INPUT_FILE = false
```
If your external markup relies on additional classes and attributes on the generated HTML elements, you might need to enable custom sanitizer policies. Gitea uses the [`bluemonday`](https://godoc.org/github.com/microcosm-cc/bluemonday) package as our HTML sanitizier. The example below will support [KaTeX](https://katex.org/) output from [`pandoc`](https://pandoc.org/).
If your external markup relies on additional classes and attributes on the generated HTML elements, you might need to enable custom sanitizer policies. Gitea uses the [`bluemonday`](https://godoc.org/github.com/microcosm-cc/bluemonday) package as our HTML sanitizer. The example below could be used to support server-side [KaTeX](https://katex.org/) rendering output from [`pandoc`](https://pandoc.org/).
```ini
[markup.sanitizer.TeX]
; Pandoc renders TeX segments as <span>s with the "math" class, optionally
; with "inline" or "display" classes depending on context.
; - note this is different from the built-in math support in our markdown parser which uses <code>
ELEMENT = span
ALLOW_ATTR = class
REGEXP = ^\s*((math(\s+|$)|inline(\s+|$)|display(\s+|$)))+

2
docs/content/doc/features/comparison.en-us.md

@ -53,6 +53,8 @@ _Symbols used in table:_
| WebAuthn (2FA) | ✓ | ✘ | ✓ | ✓ | ✓ | ✓ | ? |
| Built-in CI/CD | ✘ | ✘ | ✓ | ✓ | ✓ | ✘ | ✘ |
| Subgroups: groups within groups | [](https://github.com/go-gitea/gitea/issues/1872) | ✘ | ✘ | ✓ | ✓ | ✘ | ✓ |
| Mermaid diagrams in Markdown | ✓ | ✘ | ✓ | ✓ | ✓ | ✘ | ✘ |
| Math syntax in Markdown | ✓ | ✘ | ✓ | ✓ | ✓ | ✘ | ✘ |
## Code management

3
docs/content/page/index.en-us.md

@ -131,7 +131,8 @@ You can try it out using [the online demo](https://try.gitea.io/).
- Environment variables
- Command line options
- Multi-language support ([21 languages](https://github.com/go-gitea/gitea/tree/main/options/locale))
- [Mermaid](https://mermaidjs.github.io/) Diagram support
- [Mermaid](https://mermaidjs.github.io/) diagrams in Markdown
- Math syntax in Markdown
- Mail service
- Notifications
- Registration confirmation

2
go.mod

@ -103,6 +103,7 @@ require (
gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df
gopkg.in/ini.v1 v1.67.0
gopkg.in/yaml.v2 v2.4.0
gopkg.in/yaml.v3 v3.0.1
mvdan.cc/xurls/v2 v2.4.0
strk.kbt.io/projects/go/libravatar v0.0.0-20191008002943-06d1c002b251
xorm.io/builder v0.3.11
@ -290,7 +291,6 @@ require (
gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc // indirect
gopkg.in/cheggaaa/pb.v1 v1.0.28 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
sigs.k8s.io/yaml v1.2.0 // indirect
)

84
modules/markup/markdown/convertyaml.go

@ -0,0 +1,84 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package markdown
import (
"github.com/yuin/goldmark/ast"
east "github.com/yuin/goldmark/extension/ast"
"gopkg.in/yaml.v3"
)
func nodeToTable(meta *yaml.Node) ast.Node {
for {
if meta == nil {
return nil
}
switch meta.Kind {
case yaml.DocumentNode:
meta = meta.Content[0]
continue
default:
}
break
}
switch meta.Kind {
case yaml.MappingNode:
return mappingNodeToTable(meta)
case yaml.SequenceNode:
return sequenceNodeToTable(meta)
default:
return ast.NewString([]byte(meta.Value))
}
}
func mappingNodeToTable(meta *yaml.Node) ast.Node {
table := east.NewTable()
alignments := []east.Alignment{}
for i := 0; i < len(meta.Content); i += 2 {
alignments = append(alignments, east.AlignNone)
}
headerRow := east.NewTableRow(alignments)
valueRow := east.NewTableRow(alignments)
for i := 0; i < len(meta.Content); i += 2 {
cell := east.NewTableCell()
cell.AppendChild(cell, nodeToTable(meta.Content[i]))
headerRow.AppendChild(headerRow, cell)
if i+1 < len(meta.Content) {
cell = east.NewTableCell()
cell.AppendChild(cell, nodeToTable(meta.Content[i+1]))
valueRow.AppendChild(valueRow, cell)
}
}
table.AppendChild(table, east.NewTableHeader(headerRow))
table.AppendChild(table, valueRow)
return table
}
func sequenceNodeToTable(meta *yaml.Node) ast.Node {
table := east.NewTable()
alignments := []east.Alignment{east.AlignNone}
for _, item := range meta.Content {
row := east.NewTableRow(alignments)
cell := east.NewTableCell()
cell.AppendChild(cell, nodeToTable(item))
row.AppendChild(row, cell)
table.AppendChild(table, row)
}
return table
}
func nodeToDetails(meta *yaml.Node, icon string) ast.Node {
details := NewDetails()
summary := NewSummary()
summary.AppendChild(summary, NewIcon(icon))
details.AppendChild(details, summary)
details.AppendChild(details, nodeToTable(meta))
return details
}

15
modules/markup/markdown/goldmark.go

@ -15,7 +15,6 @@ import (
"code.gitea.io/gitea/modules/setting"
giteautil "code.gitea.io/gitea/modules/util"
meta "github.com/yuin/goldmark-meta"
"github.com/yuin/goldmark/ast"
east "github.com/yuin/goldmark/extension/ast"
"github.com/yuin/goldmark/parser"
@ -32,20 +31,12 @@ type ASTTransformer struct{}
// Transform transforms the given AST tree.
func (g *ASTTransformer) Transform(node *ast.Document, reader text.Reader, pc parser.Context) {
metaData := meta.GetItems(pc)
firstChild := node.FirstChild()
createTOC := false
ctx := pc.Get(renderContextKey).(*markup.RenderContext)
rc := &RenderConfig{
Meta: "table",
Icon: "table",
Lang: "",
}
if metaData != nil {
rc.ToRenderConfig(metaData)
metaNode := rc.toMetaNode(metaData)
rc := pc.Get(renderConfigKey).(*RenderConfig)
if rc.yamlNode != nil {
metaNode := rc.toMetaNode()
if metaNode != nil {
node.InsertBefore(node, firstChild, metaNode)
}

20
modules/markup/markdown/markdown.go

@ -14,6 +14,7 @@ import (
"code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/markup"
"code.gitea.io/gitea/modules/markup/common"
"code.gitea.io/gitea/modules/markup/markdown/math"
"code.gitea.io/gitea/modules/setting"
giteautil "code.gitea.io/gitea/modules/util"
@ -38,6 +39,7 @@ var (
isWikiKey = parser.NewContextKey()
renderMetasKey = parser.NewContextKey()
renderContextKey = parser.NewContextKey()
renderConfigKey = parser.NewContextKey()
)
type limitWriter struct {
@ -98,7 +100,7 @@ func actualRender(ctx *markup.RenderContext, input io.Reader, output io.Writer)
languageStr := string(language)
preClasses := []string{"code-block"}
if languageStr == "mermaid" {
if languageStr == "mermaid" || languageStr == "math" {
preClasses = append(preClasses, "is-loading")
}
@ -120,6 +122,9 @@ func actualRender(ctx *markup.RenderContext, input io.Reader, output io.Writer)
}
}),
),
math.NewExtension(
math.Enabled(setting.Markdown.EnableMath),
),
meta.Meta,
),
goldmark.WithParserOptions(
@ -167,7 +172,18 @@ func actualRender(ctx *markup.RenderContext, input io.Reader, output io.Writer)
log.Error("Unable to ReadAll: %v", err)
return err
}
if err := converter.Convert(giteautil.NormalizeEOL(buf), lw, parser.WithContext(pc)); err != nil {
buf = giteautil.NormalizeEOL(buf)
rc := &RenderConfig{
Meta: "table",
Icon: "table",
Lang: "",
}
buf, _ = ExtractMetadataBytes(buf, rc)
pc.Set(renderConfigKey, rc)
if err := converter.Convert(buf, lw, parser.WithContext(pc)); err != nil {
log.Error("Unable to render: %v", err)
return err
}

42
modules/markup/markdown/math/block_node.go

@ -0,0 +1,42 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package math
import "github.com/yuin/goldmark/ast"
// Block represents a display math block e.g. $$...$$ or \[...\]
type Block struct {
ast.BaseBlock
Dollars bool
Indent int
Closed bool
}
// KindBlock is the node kind for math blocks
var KindBlock = ast.NewNodeKind("MathBlock")
// NewBlock creates a new math Block
func NewBlock(dollars bool, indent int) *Block {
return &Block{
Dollars: dollars,
Indent: indent,
}
}
// Dump dumps the block to a string
func (n *Block) Dump(source []byte, level int) {
m := map[string]string{}
ast.DumpHelper(n, source, level, m, nil)
}
// Kind returns KindBlock for math Blocks
func (n *Block) Kind() ast.NodeKind {
return KindBlock
}
// IsRaw returns true as this block should not be processed further
func (n *Block) IsRaw() bool {
return true
}

123
modules/markup/markdown/math/block_parser.go

@ -0,0 +1,123 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package math
import (
"bytes"
"github.com/yuin/goldmark/ast"
"github.com/yuin/goldmark/parser"
"github.com/yuin/goldmark/text"
"github.com/yuin/goldmark/util"
)
type blockParser struct {
parseDollars bool
}
// NewBlockParser creates a new math BlockParser
func NewBlockParser(parseDollarBlocks bool) parser.BlockParser {
return &blockParser{
parseDollars: parseDollarBlocks,
}
}
// Open parses the current line and returns a result of parsing.
func (b *blockParser) Open(parent ast.Node, reader text.Reader, pc parser.Context) (ast.Node, parser.State) {
line, segment := reader.PeekLine()
pos := pc.BlockOffset()
if pos == -1 || len(line[pos:]) < 2 {
return nil, parser.NoChildren
}
dollars := false
if b.parseDollars && line[pos] == '$' && line[pos+1] == '$' {
dollars = true
} else if line[pos] != '\\' || line[pos+1] != '[' {
return nil, parser.NoChildren
}
node := NewBlock(dollars, pos)
// Now we need to check if the ending block is on the segment...
endBytes := []byte{'\\', ']'}
if dollars {
endBytes = []byte{'$', '$'}
}
idx := bytes.Index(line[pos+2:], endBytes)
if idx >= 0 {
segment.Stop = segment.Start + idx + 2
reader.Advance(segment.Len() - 1)
segment.Start += 2
node.Lines().Append(segment)
node.Closed = true
return node, parser.Close | parser.NoChildren
}
reader.Advance(segment.Len() - 1)
segment.Start += 2
node.Lines().Append(segment)
return node, parser.NoChildren
}
// Continue parses the current line and returns a result of parsing.
func (b *blockParser) Continue(node ast.Node, reader text.Reader, pc parser.Context) parser.State {
block := node.(*Block)
if block.Closed {
return parser.Close
}
line, segment := reader.PeekLine()
w, pos := util.IndentWidth(line, 0)
if w < 4 {
if block.Dollars {
i := pos
for ; i < len(line) && line[i] == '$'; i++ {
}
length := i - pos
if length >= 2 && util.IsBlank(line[i:]) {
reader.Advance(segment.Stop - segment.Start - segment.Padding)
block.Closed = true
return parser.Close
}
} else if len(line[pos:]) > 1 && line[pos] == '\\' && line[pos+1] == ']' && util.IsBlank(line[pos+2:]) {
reader.Advance(segment.Stop - segment.Start - segment.Padding)
block.Closed = true
return parser.Close
}
}
pos, padding := util.IndentPosition(line, 0, block.Indent)
seg := text.NewSegmentPadding(segment.Start+pos, segment.Stop, padding)
node.Lines().Append(seg)
reader.AdvanceAndSetPadding(segment.Stop-segment.Start-pos-1, padding)
return parser.Continue | parser.NoChildren
}
// Close will be called when the parser returns Close.
func (b *blockParser) Close(node ast.Node, reader text.Reader, pc parser.Context) {
// noop
}
// CanInterruptParagraph returns true if the parser can interrupt paragraphs,
// otherwise false.
func (b *blockParser) CanInterruptParagraph() bool {
return true
}
// CanAcceptIndentedLine returns true if the parser can open new node when
// the given line is being indented more than 3 spaces.
func (b *blockParser) CanAcceptIndentedLine() bool {
return false
}
// Trigger returns a list of characters that triggers Parse method of
// this parser.
// If Trigger returns a nil, Open will be called with any lines.
//
// We leave this as nil as our parse method is quick enough
func (b *blockParser) Trigger() []byte {
return nil
}

43
modules/markup/markdown/math/block_renderer.go

@ -0,0 +1,43 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package math
import (
gast "github.com/yuin/goldmark/ast"
"github.com/yuin/goldmark/renderer"
"github.com/yuin/goldmark/util"
)
// BlockRenderer represents a renderer for math Blocks
type BlockRenderer struct{}
// NewBlockRenderer creates a new renderer for math Blocks
func NewBlockRenderer() renderer.NodeRenderer {
return &BlockRenderer{}
}
// RegisterFuncs registers the renderer for math Blocks
func (r *BlockRenderer) RegisterFuncs(reg renderer.NodeRendererFuncRegisterer) {
reg.Register(KindBlock, r.renderBlock)
}
func (r *BlockRenderer) writeLines(w util.BufWriter, source []byte, n gast.Node) {
l := n.Lines().Len()
for i := 0; i < l; i++ {
line := n.Lines().At(i)
_, _ = w.Write(util.EscapeHTML(line.Value(source)))
}
}
func (r *BlockRenderer) renderBlock(w util.BufWriter, source []byte, node gast.Node, entering bool) (gast.WalkStatus, error) {
n := node.(*Block)
if entering {
_, _ = w.WriteString(`<pre class="code-block is-loading"><code class="chroma language-math display">`)
r.writeLines(w, source, n)
} else {
_, _ = w.WriteString(`</code></pre>` + "\n")
}
return gast.WalkContinue, nil
}

49
modules/markup/markdown/math/inline_node.go

@ -0,0 +1,49 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package math
import (
"github.com/yuin/goldmark/ast"
"github.com/yuin/goldmark/util"
)
// Inline represents inline math e.g. $...$ or \(...\)
type Inline struct {
ast.BaseInline
}
// Inline implements Inline.Inline.
func (n *Inline) Inline() {}
// IsBlank returns if this inline node is empty
func (n *Inline) IsBlank(source []byte) bool {
for c := n.FirstChild(); c != nil; c = c.NextSibling() {
text := c.(*ast.Text).Segment
if !util.IsBlank(text.Value(source)) {
return false
}
}
return true
}
// Dump renders this inline math as debug
func (n *Inline) Dump(source []byte, level int) {
ast.DumpHelper(n, source, level, nil, nil)
}
// KindInline is the kind for math inline
var KindInline = ast.NewNodeKind("MathInline")
// Kind returns KindInline
func (n *Inline) Kind() ast.NodeKind {
return KindInline
}
// NewInline creates a new ast math inline node
func NewInline() *Inline {
return &Inline{
BaseInline: ast.BaseInline{},
}
}

99
modules/markup/markdown/math/inline_parser.go

@ -0,0 +1,99 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package math
import (
"bytes"
"github.com/yuin/goldmark/ast"
"github.com/yuin/goldmark/parser"
"github.com/yuin/goldmark/text"
)
type inlineParser struct {
start []byte
end []byte
}
var defaultInlineDollarParser = &inlineParser{
start: []byte{'$'},
end: []byte{'$'},
}
// NewInlineDollarParser returns a new inline parser
func NewInlineDollarParser() parser.InlineParser {
return defaultInlineDollarParser
}
var defaultInlineBracketParser = &inlineParser{
start: []byte{'\\', '('},
end: []byte{'\\', ')'},
}
// NewInlineDollarParser returns a new inline parser
func NewInlineBracketParser() parser.InlineParser {
return defaultInlineBracketParser
}
// Trigger triggers this parser on $
func (parser *inlineParser) Trigger() []byte {
return parser.start[0:1]
}
func isAlphanumeric(b byte) bool {
// Github only cares about 0-9A-Za-z
return (b >= '0' && b <= '9') || (b >= 'A' && b <= 'Z') || (b >= 'a' && b <= 'z')
}
// Parse parses the current line and returns a result of parsing.
func (parser *inlineParser) Parse(parent ast.Node, block text.Reader, pc parser.Context) ast.Node {
line, _ := block.PeekLine()
opener := bytes.Index(line, parser.start)
if opener < 0 {
return nil
}
if opener != 0 && isAlphanumeric(line[opener-1]) {
return nil
}
opener += len(parser.start)
ender := bytes.Index(line[opener:], parser.end)
if ender < 0 {
return nil
}
if len(line) > opener+ender+len(parser.end) && isAlphanumeric(line[opener+ender+len(parser.end)]) {
return nil
}
block.Advance(opener)
_, pos := block.Position()
node := NewInline()
segment := pos.WithStop(pos.Start + ender)
node.AppendChild(node, ast.NewRawTextSegment(segment))
block.Advance(ender + len(parser.end))
trimBlock(node, block)
return node
}
func trimBlock(node *Inline, block text.Reader) {
if node.IsBlank(block.Source()) {
return
}
// trim first space and last space
first := node.FirstChild().(*ast.Text)
if !(!first.Segment.IsEmpty() && block.Source()[first.Segment.Start] == ' ') {
return
}
last := node.LastChild().(*ast.Text)
if !(!last.Segment.IsEmpty() && block.Source()[last.Segment.Stop-1] == ' ') {
return
}
first.Segment = first.Segment.WithStart(first.Segment.Start + 1)
last.Segment = last.Segment.WithStop(last.Segment.Stop - 1)
}

47
modules/markup/markdown/math/inline_renderer.go

@ -0,0 +1,47 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package math
import (
"bytes"
"github.com/yuin/goldmark/ast"
"github.com/yuin/goldmark/renderer"
"github.com/yuin/goldmark/util"
)
// InlineRenderer is an inline renderer
type InlineRenderer struct{}
// NewInlineRenderer returns a new renderer for inline math
func NewInlineRenderer() renderer.NodeRenderer {
return &InlineRenderer{}
}
func (r *InlineRenderer) renderInline(w util.BufWriter, source []byte, n ast.Node, entering bool) (ast.WalkStatus, error) {
if entering {
_, _ = w.WriteString(`<code class="language-math is-loading">`)
for c := n.FirstChild(); c != nil; c = c.NextSibling() {
segment := c.(*ast.Text).Segment
value := util.EscapeHTML(segment.Value(source))
if bytes.HasSuffix(value, []byte("\n")) {
_, _ = w.Write(value[:len(value)-1])
if c != n.LastChild() {
_, _ = w.Write([]byte(" "))
}
} else {
_, _ = w.Write(value)
}
}
return ast.WalkSkipChildren, nil
}
_, _ = w.WriteString(`</code>`)
return ast.WalkContinue, nil
}
// RegisterFuncs registers the renderer for inline math nodes
func (r *InlineRenderer) RegisterFuncs(reg renderer.NodeRendererFuncRegisterer) {
reg.Register(KindInline, r.renderInline)
}

108
modules/markup/markdown/math/math.go

@ -0,0 +1,108 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package math
import (
"github.com/yuin/goldmark"
"github.com/yuin/goldmark/parser"
"github.com/yuin/goldmark/renderer"
"github.com/yuin/goldmark/util"
)
// Extension is a math extension
type Extension struct {
enabled bool
parseDollarInline bool
parseDollarBlock bool
}
// Option is the interface Options should implement
type Option interface {
SetOption(e *Extension)
}
type extensionFunc func(e *Extension)
func (fn extensionFunc) SetOption(e *Extension) {
fn(e)
}
// Enabled enables or disables this extension
func Enabled(enable ...bool) Option {
value := true
if len(enable) > 0 {
value = enable[0]
}
return extensionFunc(func(e *Extension) {
e.enabled = value
})
}
// WithInlineDollarParser enables or disables the parsing of $...$
func WithInlineDollarParser(enable ...bool) Option {
value := true
if len(enable) > 0 {
value = enable[0]
}
return extensionFunc(func(e *Extension) {
e.parseDollarInline = value
})
}
// WithBlockDollarParser enables or disables the parsing of $$...$$
func WithBlockDollarParser(enable ...bool) Option {
value := true
if len(enable) > 0 {
value = enable[0]
}
return extensionFunc(func(e *Extension) {
e.parseDollarBlock = value
})
}
// Math represents a math extension with default rendered delimiters
var Math = &Extension{
enabled: true,
parseDollarBlock: true,
parseDollarInline: true,
}
// NewExtension creates a new math extension with the provided options
func NewExtension(opts ...Option) *Extension {
r := &Extension{
enabled: true,
parseDollarBlock: true,
parseDollarInline: true,
}
for _, o := range opts {
o.SetOption(r)
}
return r
}
// Extend extends goldmark with our parsers and renderers
func (e *Extension) Extend(m goldmark.Markdown) {
if !e.enabled {
return
}
m.Parser().AddOptions(parser.WithBlockParsers(
util.Prioritized(NewBlockParser(e.parseDollarBlock), 701),
))
inlines := []util.PrioritizedValue{
util.Prioritized(NewInlineBracketParser(), 501),
}
if e.parseDollarInline {
inlines = append(inlines, util.Prioritized(NewInlineDollarParser(), 501))
}
m.Parser().AddOptions(parser.WithInlineParsers(inlines...))
m.Renderer().AddOptions(renderer.WithNodeRenderers(
util.Prioritized(NewBlockRenderer(), 501),
util.Prioritized(NewInlineRenderer(), 502),
))
}

96
modules/markup/markdown/meta.go

@ -5,47 +5,101 @@
package markdown
import (
"bytes"
"errors"
"strings"
"unicode"
"unicode/utf8"
"gopkg.in/yaml.v2"
"code.gitea.io/gitea/modules/log"
"gopkg.in/yaml.v3"
)
func isYAMLSeparator(line string) bool {
line = strings.TrimSpace(line)
for i := 0; i < len(line); i++ {
if line[i] != '-' {
func isYAMLSeparator(line []byte) bool {
idx := 0
for ; idx < len(line); idx++ {
if line[idx] >= utf8.RuneSelf {
r, sz := utf8.DecodeRune(line[idx:])
if !unicode.IsSpace(r) {
return false
}
idx += sz
continue
}
if line[idx] != ' ' {
break
}
}
dashCount := 0
for ; idx < len(line); idx++ {
if line[idx] != '-' {
break
}
dashCount++
}
if dashCount < 3 {
return false
}
for ; idx < len(line); idx++ {
if line[idx] >= utf8.RuneSelf {
r, sz := utf8.DecodeRune(line[idx:])
if !unicode.IsSpace(r) {
return false
}
idx += sz
continue
}
if line[idx] != ' ' {
return false
}
}
return len(line) > 2
return true
}
// ExtractMetadata consumes a markdown file, parses YAML frontmatter,
// and returns the frontmatter metadata separated from the markdown content
func ExtractMetadata(contents string, out interface{}) (string, error) {
var front, body []string
lines := strings.Split(contents, "\n")
for idx, line := range lines {
if idx == 0 {
// First line has to be a separator
if !isYAMLSeparator(line) {
return "", errors.New("frontmatter must start with a separator line")
}
continue
body, err := ExtractMetadataBytes([]byte(contents), out)
return string(body), err
}
// ExtractMetadata consumes a markdown file, parses YAML frontmatter,
// and returns the frontmatter metadata separated from the markdown content
func ExtractMetadataBytes(contents []byte, out interface{}) ([]byte, error) {
var front, body []byte
start, end := 0, len(contents)
idx := bytes.IndexByte(contents[start:], '\n')
if idx >= 0 {
end = start + idx
}
line := contents[start:end]
if !isYAMLSeparator(line) {
return contents, errors.New("frontmatter must start with a separator line")
}
frontMatterStart := end + 1
for start = frontMatterStart; start < len(contents); start = end + 1 {
end = len(contents)
idx := bytes.IndexByte(contents[start:], '\n')
if idx >= 0 {
end = start + idx
}
line := contents[start:end]
if isYAMLSeparator(line) {
front, body = lines[1:idx], lines[idx+1:]
front = contents[frontMatterStart:start]
body = contents[end+1:]
break
}
}
if len(front) == 0 {
return "", errors.New("could not determine metadata")
return contents, errors.New("could not determine metadata")
}
if err := yaml.Unmarshal([]byte(strings.Join(front, "\n")), out); err != nil {
return "", err
log.Info("%s", string(front))
if err := yaml.Unmarshal(front, out); err != nil {
return contents, err
}
return strings.Join(body, "\n"), nil
return body, nil
}

32
modules/markup/markdown/meta_test.go

@ -56,6 +56,38 @@ func TestExtractMetadata(t *testing.T) {
})
}
func TestExtractMetadataBytes(t *testing.T) {
t.Run("ValidFrontAndBody", func(t *testing.T) {
var meta structs.IssueTemplate
body, err := ExtractMetadataBytes([]byte(fmt.Sprintf("%s\n%s\n%s\n%s", sepTest, frontTest, sepTest, bodyTest)), &meta)
assert.NoError(t, err)
assert.Equal(t, bodyTest, body)
assert.Equal(t, metaTest, meta)
assert.True(t, validateMetadata(meta))
})
t.Run("NoFirstSeparator", func(t *testing.T) {
var meta structs.IssueTemplate
_, err := ExtractMetadataBytes([]byte(fmt.Sprintf("%s\n%s\n%s", frontTest, sepTest, bodyTest)), &meta)
assert.Error(t, err)
})
t.Run("NoLastSeparator", func(t *testing.T) {
var meta structs.IssueTemplate
_, err := ExtractMetadataBytes([]byte(fmt.Sprintf("%s\n%s\n%s", sepTest, frontTest, bodyTest)), &meta)
assert.Error(t, err)
})
t.Run("NoBody", func(t *testing.T) {
var meta structs.IssueTemplate
body, err := ExtractMetadataBytes([]byte(fmt.Sprintf("%s\n%s\n%s", sepTest, frontTest, sepTest)), &meta)
assert.NoError(t, err)
assert.Equal(t, "", body)
assert.Equal(t, metaTest, meta)
assert.True(t, validateMetadata(meta))
})
}
var (
sepTest = "-----"
frontTest = `name: Test

219
modules/markup/markdown/renderconfig.go

@ -5,159 +5,114 @@
package markdown
import (
"fmt"
"strings"
"code.gitea.io/gitea/modules/log"
"github.com/yuin/goldmark/ast"
east "github.com/yuin/goldmark/extension/ast"
"gopkg.in/yaml.v2"
"gopkg.in/yaml.v3"
)
// RenderConfig represents rendering configuration for this file
type RenderConfig struct {
Meta string
Icon string
TOC bool
Lang string
Meta string
Icon string
TOC bool
Lang string
yamlNode *yaml.Node
}
// ToRenderConfig converts a yaml.MapSlice to a RenderConfig
func (rc *RenderConfig) ToRenderConfig(meta yaml.MapSlice) {
if meta == nil {
return
}
found := false
var giteaMetaControl yaml.MapItem
for _, item := range meta {
strKey, ok := item.Key.(string)
if !ok {
continue
}
strKey = strings.TrimSpace(strings.ToLower(strKey))
switch strKey {
case "gitea":
giteaMetaControl = item
found = true
case "include_toc":
val, ok := item.Value.(bool)
if !ok {
continue
}
rc.TOC = val
case "lang":
val, ok := item.Value.(string)
if !ok {
continue
}
val = strings.TrimSpace(val)
if len(val) == 0 {
continue
}
rc.Lang = val
// UnmarshalYAML implement yaml.v3 UnmarshalYAML
func (rc *RenderConfig) UnmarshalYAML(value *yaml.Node) error {
if rc == nil {
rc = &RenderConfig{
Meta: "table",
Icon: "table",
Lang: "",
}
}
rc.yamlNode = value
if found {
switch v := giteaMetaControl.Value.(type) {
case string:
switch v {
case "none":
rc.Meta = "none"
case "table":
rc.Meta = "table"
default: // "details"
rc.Meta = "details"
}
case yaml.MapSlice:
for _, item := range v {
strKey, ok := item.Key.(string)
if !ok {
continue
}
strKey = strings.TrimSpace(strings.ToLower(strKey))
switch strKey {
case "meta":
val, ok := item.Value.(string)
if !ok {
continue
}
switch strings.TrimSpace(strings.ToLower(val)) {
case "none":
rc.Meta = "none"
case "table":
rc.Meta = "table"
default: // "details"
rc.Meta = "details"
}
case "details_icon":
val, ok := item.Value.(string)
if !ok {
continue
}
rc.Icon = strings.TrimSpace(strings.ToLower(val))
case "include_toc":
val, ok := item.Value.(bool)
if !ok {
continue
}
rc.TOC = val
case "lang":
val, ok := item.Value.(string)
if !ok {
continue
}
val = strings.TrimSpace(val)
if len(val) == 0 {
continue
}
rc.Lang = val
}
}
}
type basicRenderConfig struct {
Gitea *yaml.Node `yaml:"gitea"`
TOC bool `yaml:"include_toc"`
Lang string `yaml:"lang"`
}
}
func (rc *RenderConfig) toMetaNode(meta yaml.MapSlice) ast.Node {
switch rc.Meta {
case "table":
return metaToTable(meta)
case "details":
return metaToDetails(meta, rc.Icon)
default:
var basic basicRenderConfig
err := value.Decode(&basic)
if err != nil {
return err
}
if basic.Lang != "" {
rc.Lang = basic.Lang
}
rc.TOC = basic.TOC
if basic.Gitea == nil {
return nil
}
}
func metaToTable(meta yaml.MapSlice) ast.Node {
table := east.NewTable()
alignments := []east.Alignment{}
for range meta {
alignments = append(alignments, east.AlignNone)
var control *string
if err := basic.Gitea.Decode(&control); err == nil && control != nil {
log.Info("control %v", control)
switch strings.TrimSpace(strings.ToLower(*control)) {
case "none":
rc.Meta = "none"
case "table":
rc.Meta = "table"
default: // "details"
rc.Meta = "details"
}
return nil
}
row := east.NewTableRow(alignments)
for _, item := range meta {
cell := east.NewTableCell()
cell.AppendChild(cell, ast.NewString([]byte(fmt.Sprintf("%v", item.Key))))
row.AppendChild(row, cell)
type giteaControl struct {
Meta string `yaml:"meta"`
Icon string `yaml:"details_icon"`
TOC *yaml.Node `yaml:"include_toc"`
Lang string `yaml:"lang"`
}
table.AppendChild(table, east.NewTableHeader(row))
row = east.NewTableRow(alignments)
for _, item := range meta {
cell := east.NewTableCell()
cell.AppendChild(cell, ast.NewString([]byte(fmt.Sprintf("%v", item.Value))))
row.AppendChild(row, cell)
var controlStruct *giteaControl
if err := basic.Gitea.Decode(controlStruct); err != nil || controlStruct == nil {
return err
}
table.AppendChild(table, row)
return table
}
func metaToDetails(meta yaml.MapSlice, icon string) ast.Node {
details := NewDetails()
summary := NewSummary()
summary.AppendChild(summary, NewIcon(icon))
details.AppendChild(details, summary)
details.AppendChild(details, metaToTable(meta))
switch strings.TrimSpace(strings.ToLower(controlStruct.Meta)) {
case "none":
rc.Meta = "none"
case "table":
rc.Meta = "table"
default: // "details"
rc.Meta = "details"
}
rc.Icon = strings.TrimSpace(strings.ToLower(controlStruct.Icon))
if controlStruct.Lang != "" {
rc.Lang = controlStruct.Lang
}
var toc bool
if err := controlStruct.TOC.Decode(&toc); err == nil {
rc.TOC = toc
}
return nil
}
return details
func (rc *RenderConfig) toMetaNode() ast.Node {
if rc.yamlNode == nil {
return nil
}
switch rc.Meta {
case "table":
return nodeToTable(rc.yamlNode)
case "details":
return nodeToDetails(rc.yamlNode, rc.Icon)
default:
return nil
}
}

162
modules/markup/markdown/renderconfig_test.go

@ -0,0 +1,162 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package markdown
import (
"testing"
"gopkg.in/yaml.v3"
)
func TestRenderConfig_UnmarshalYAML(t *testing.T) {
tests := []struct {
name string
expected *RenderConfig
args string
}{
{
"empty", &RenderConfig{
Meta: "table",
Icon: "table",
Lang: "",
}, "",
},
{
"lang", &RenderConfig{
Meta: "table",
Icon: "table",
Lang: "test",
}, "lang: test",
},
{
"metatable", &RenderConfig{
Meta: "table",
Icon: "table",
Lang: "",
}, "gitea: table",
},
{
"metanone", &RenderConfig{
Meta: "none",
Icon: "table",
Lang: "",
}, "gitea: none",
},
{
"metadetails", &RenderConfig{
Meta: "details",
Icon: "table",
Lang: "",
}, "gitea: details",
},
{
"metawrong", &RenderConfig{
Meta: "details",
Icon: "table",
Lang: "",
}, "gitea: wrong",
},
{
"toc", &RenderConfig{
TOC: true,
Meta: "table",
Icon: "table",
Lang: "",
}, "include_toc: true",
},
{
"tocfalse", &RenderConfig{
TOC: false,
Meta: "table",
Icon: "table",
Lang: "",
}, "include_toc: false",
},
{
"toclang", &RenderConfig{
Meta: "table",
Icon: "table",
TOC: true,
Lang: "testlang",
}, `
include_toc: true
lang: testlang
`,
},
{
"complexlang", &RenderConfig{
Meta: "table",
Icon: "table",
Lang: "testlang",
}, `
gitea:
lang: testlang
`,
},
{
"complexlang2", &RenderConfig{
Meta: "table",
Icon: "table",
Lang: "testlang",
}, `
lang: notright
gitea:
lang: testlang
`,
},
{
"complexlang", &RenderConfig{
Meta: "table",
Icon: "table",
Lang: "testlang",
}, `
gitea:
lang: testlang
`,
},
{
"complex2", &RenderConfig{
Lang: "two",
Meta: "table",
TOC: true,
Icon: "smiley",
}, `
lang: one
include_toc: true
gitea:
details_icon: smiley
meta: table
include_toc: true
lang: two
`,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := &RenderConfig{
Meta: "table",
Icon: "table",
Lang: "",
}
if err := yaml.Unmarshal([]byte(tt.args), got); err != nil {
t.Errorf("RenderConfig.UnmarshalYAML() error = %v", err)
return
}
if got.Meta != tt.expected.Meta {
t.Errorf("Meta Expected %s Got %s", tt.expected.Meta, got.Meta)
}
if got.Icon != tt.expected.Icon {
t.Errorf("Icon Expected %s Got %s", tt.expected.Icon, got.Icon)
}
if got.Lang != tt.expected.Lang {
t.Errorf("Lang Expected %s Got %s", tt.expected.Lang, got.Lang)
}
if got.TOC != tt.expected.TOC {
t.Errorf("TOC Expected %t Got %t", tt.expected.TOC, got.TOC)
}
})
}
}

4
modules/markup/sanitizer.go

@ -56,7 +56,7 @@ func createDefaultPolicy() *bluemonday.Policy {
policy.AllowAttrs("class").Matching(regexp.MustCompile(`^code-block( is-loading)?$`)).OnElements("pre")
// For Chroma markdown plugin
policy.AllowAttrs("class").Matching(regexp.MustCompile(`^(chroma )?language-[\w-]+$`)).OnElements("code")
policy.AllowAttrs("class").Matching(regexp.MustCompile(`^(chroma )?language-[\w-]+( display)?( is-loading)?$`)).OnElements("code")
// Checkboxes
policy.AllowAttrs("type").Matching(regexp.MustCompile(`^checkbox$`)).OnElements("input")