highlighter: Use clip_offset to ensure UTF-8 char boundaries (#1317)

## Problem

SyntaxHighlighter panics when highlighting text containing multi-byte
UTF-8 characters (Chinese, Japanese, emoji, etc).

**Error**: `thread 'main' panicked at
ropey-2.0.0-beta.1/src/rope.rs:694:59`

**Root Cause**: tree-sitter may return byte offsets that fall in the
middle of multi-byte UTF-8 characters. When `Rope::slice()` receives
such offsets, it checks `is_char_boundary()` and panics with
`NonCharBoundary` error.

## Solution

Use `RopeExt::clip_offset()` to adjust byte offsets to the nearest char
boundaries before calling `slice()`.

**Changes**:
- Line 395-398: Clip offsets in main highlighting loop
- Line 441-443: Clip offsets when extracting injection content  
- Line 478-479: Added explanatory comment for injection highlighting
- Added imports for `RopeExt` and `Bias`

**Safety**:
- `Bias::Left` for start offset ensures we don't skip the beginning of a
character
- `Bias::Right` for end offset ensures we include the full character
- The adjustment is minimal (at most 3 bytes for UTF-8)
- Uses existing project API (`RopeExt::clip_offset`)

## Testing

Tested with:
-  Chinese text: "你好世界"、"**加粗中文**"
-  Japanese text: "こんにちは"、"日本語"
-  Emoji: "😀🎉🚀"
-  Markdown syntax highlighting with CJK characters
-  Code blocks with mixed languages

All tests pass without panics. Syntax highlighting works correctly. No
performance degradation observed.

## AI Assistance

🤖 This fix was developed with AI assistance (Claude). The solution
approach (using `clip_offset`) was identified through code analysis and
testing. The AI analyzed:
- ropey source code to understand the panic condition
- Existing usage of `clip_offset` in the codebase
- Tree-sitter byte offset behavior with UTF-8

All code has been reviewed and tested by humans, and further validated
by Gemini AI.

## Checklist

- [x] Follows existing code style
- [x] One PR does one thing (UTF-8 panic fix only)
- [x] All manual tests pass
- [x] Tested with real multi-byte UTF-8 content
- [x] No breaking changes
- [x] No performance regression
This commit is contained in:
Xu Desheng 2025-10-02 00:14:07 -04:00 committed by GitHub
parent de4a7a7026
commit 8c6cde3719
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -1,4 +1,5 @@
use crate::highlighter::{HighlightTheme, LanguageRegistry};
use crate::input::RopeExt;
use anyhow::{anyhow, Context, Result};
use gpui::{HighlightStyle, SharedString};
@ -9,6 +10,7 @@ use std::{
ops::Range,
usize,
};
use sum_tree::Bias;
use tree_sitter::{
InputEdit, Node, Parser, Point, Query, QueryCursor, QueryMatch, StreamingIterator, Tree,
};
@ -433,14 +435,16 @@ impl SyntaxHighlighter {
injection_language: &str,
node: Node,
) -> Vec<(Range<usize>, String)> {
let start_offset = node.start_byte();
let end_offset = node.end_byte();
// Ensure byte offsets are on char boundaries for UTF-8 safety
let start_offset = self.text.clip_offset(node.start_byte(), Bias::Left);
let end_offset = self.text.clip_offset(node.end_byte(), Bias::Right);
let mut cache = vec![];
let Some(query) = &self.injection_queries.get(injection_language) else {
return cache;
};
let content = self.text.slice(node.byte_range());
let content = self.text.slice(start_offset..end_offset);
if content.len() == 0 {
return cache;
};