Add unicode encoding and decoding functionality

[vkusrorof]

Recommendations on requirements for {{unicode(encode,decode)}} #6177

① This verification only uses the "Chinese" Unicode for testing and simply decodes Chinese using the Go language.
② The complete Unicode dictionary was found on the Internet: https://www.unicode.org/versions/Unicode16.0.0/#Components
③ The ultimate goal is to achieve the function that when Unicode is matched, it can be decoded (in addition, it is hoped that nuclei can specify the text for encoding during verification).

I wrote the exploration script, but some urls appear unicode encoding characters. For example, baidu.com shows "\u767e\u5ea6\u4e00\u4e0b\uff0c\u4f60\u5c31\u77e5\u9053".

I usually use unicode after extracting "Chinese" validate (conversion website: http://www.jsons.cn/unicode)

yaml:
id: alive-check-20250328

info:
name: alive-check
author: alive-check
severity: info
description: status test

http:

• raw:

  • |
    GET / HTTP/1.1
    Host: {{Hostname}}

  matchers-condition: and
  matchers:

  • type: status
    status:
    • 200
      extractors:
  • type: regex
    name: title
    group: 1
    regex:
    • "<title>(.*?)</title>"
  • type: regex
    group: 1
    regex:
    • 'top.location.replace("([^"]+)")'

After consulting, I found that the go language supports unicode decoding. This code tests reading the local "unicodelist.txt" file and performing decoding.

unicodelist.txt sample:

\u767e\u5ea6\u4e00\u4e0b\u006f\u006b\uff0c\u4f60\u5c31\u77e5\u9053\uff0c\ua\u662f\u0020\u0061\u0061
\u767e\u5ea6\u4e00\u4e0b\u006f\u006b
\u767e\u5ea6\u4e00\u4e0b\u006f\u006b
\u006f\u006b

main.go run result:

go test code:

package main

import (
	"bufio"
	"bytes"
	"fmt"
	"os"
	"regexp"
	"strconv"
	"strings"
)

// Fix the broken escape of \u (such as \ua → line break + \u)
func fixBrokenUnicode(data string) string {
	var result strings.Builder
	i := 0
	for i < len(data) {
		if strings.HasPrefix(data[i:], `\u`) {
			// Attempt to take four hexadecimal characters
			end := i + 6
			if end <= len(data) {
				hex := data[i+2 : end]
				if matched, _ := regexp.MatchString(`^[0-9a-fA-F]{4}$`, hex); matched {
					result.WriteString(data[i:end])
					i = end
					continue
				}
			}
			// Illegal \u escape, skipping the current \u and up to 4 characters following it
			j := i + 2
			for j < len(data) && j-i < 6 {
				if !((data[j] >= '0' && data[j] <= '9') ||
					(data[j] >= 'a' && data[j] <= 'f') ||
					(data[j] >= 'A' && data[j] <= 'F')) {
					break
				}
				j++
			}
			result.WriteString("\n")
			i = j // Skip the part of illegal escape
		} else {
			result.WriteByte(data[i])
			i++
		}
	}
	return result.String()
}

// Decode \uXXXX or \UXXXXXXXX
func EscapeUnicode(data []byte) []byte {
	re := regexp.MustCompile(`(\\u[0-9a-fA-F]{4}|\\U[0-9a-fA-F]{8})+`)
	for _, match := range re.FindAll(data, -1) {
		str, err := strconv.Unquote(`"` + string(match) + `"`)
		if err == nil {
			data = bytes.ReplaceAll(data, match, []byte(str))
		}
	}
	return data
}

func main() {
	file, err := os.Open("unicodelist.txt")
	if err != nil {
		fmt.Println("Failed to open the file:", err)
		return
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)
	lineNum := 1
	for scanner.Scan() {
		line := scanner.Text()

		// Fix illegal Unicode escape (line breaks)
		fixedLine := fixBrokenUnicode(line)

		// Decode as UTF-8
		decoded := EscapeUnicode([]byte(fixedLine))

		fmt.Printf("line %d : %s\n", lineNum, string(decoded))
		lineNum++
	}

	if err := scanner.Err(); err != nil {
		fmt.Println("Error in reading the file:", err)
	}
}

[vkusrorof]

@dogancanbakir Thank you very much for your attention.

[vkusrorof]

Dear Administrator(@dogancanbakir ), although the running results of Nuclei show Unicode encoding (the debug output displays normal Chinese characters), the extractors only output Unicode encoding when extracting the title of the returned page.

To address this issue, I have developed a new solution (and I would also like to express my appreciation for the robust functionality provided by Nuclei). By performing Base64 encoding on the content to be extracted, I can subsequently perform batch decoding using Python after data export, thereby successfully retrieving titles in Chinese characters. Even in cases where the webpage uses GB2312 encoding, Python is capable of automatically detecting and decoding the content correctly.

nuclei -t final-head-extractor-20250805.yaml -duc -nm -o test.txt -l list.txt

info:
  name: Final Page <head> Extractor with Base64
  author: optimized-by-chatgpt
  severity: info
  description: |
    Follows redirects to final landing page and extracts <head> content (base64-encoded).

http:
  - method: GET
    path:
      - "{{BaseURL}}"

    redirects: true
    max-redirects: 5
    max-size: 200000

    headers:
      User-Agent: Mozilla/5.0 (Linux; Android 10; Mobile) AppleWebKit/537.36 Chrome/115.0.0.0 Mobile Safari/537.36
      Accept-Encoding: gzip
      Connection: close

    matchers:
      - type: status
        status:
          - 200

    extractors:
      - type: regex
        name: head_raw
        internal: true
        group: 1
        regex:
          - (?is)<head[^>]*>(.*?)</head>

      - type: dsl
        name: head_base64
        dsl:
          - base64(head_raw)

The following illustrates the process in which Python reads the output results from Nuclei, performs decoding operations—including Base64 decoding, HTML entity decoding, and conversion from GB2312 to UTF-8 encoding—and subsequently conducts local rule matching.

[dogancanbakir]

@vkusrorof It's great to see that you were able to solve this with templating system without requiring DSL funcs. Thanks for sharing! I'll still keep this issue open and possibly implement it.

[vkusrorof]

@vkusrorof It's great to see that you were able to solve this with templating system without requiring DSL funcs. Thanks for sharing! I'll still keep this issue open and possibly implement it.

Thank you for your interest in the suggested feature. Wishing you all the best and good health!