8c6cde3719
## Problem SyntaxHighlighter panics when highlighting text containing multi-byte UTF-8 characters (Chinese, Japanese, emoji, etc). **Error**: `thread 'main' panicked at ropey-2.0.0-beta.1/src/rope.rs:694:59` **Root Cause**: tree-sitter may return byte offsets that fall in the middle of multi-byte UTF-8 characters. When `Rope::slice()` receives such offsets, it checks `is_char_boundary()` and panics with `NonCharBoundary` error. ## Solution Use `RopeExt::clip_offset()` to adjust byte offsets to the nearest char boundaries before calling `slice()`. **Changes**: - Line 395-398: Clip offsets in main highlighting loop - Line 441-443: Clip offsets when extracting injection content - Line 478-479: Added explanatory comment for injection highlighting - Added imports for `RopeExt` and `Bias` **Safety**: - `Bias::Left` for start offset ensures we don't skip the beginning of a character - `Bias::Right` for end offset ensures we include the full character - The adjustment is minimal (at most 3 bytes for UTF-8) - Uses existing project API (`RopeExt::clip_offset`) ## Testing Tested with: - ✅ Chinese text: "你好世界"、"**加粗中文**" - ✅ Japanese text: "こんにちは"、"日本語" - ✅ Emoji: "😀🎉✨🚀" - ✅ Markdown syntax highlighting with CJK characters - ✅ Code blocks with mixed languages All tests pass without panics. Syntax highlighting works correctly. No performance degradation observed. ## AI Assistance 🤖 This fix was developed with AI assistance (Claude). The solution approach (using `clip_offset`) was identified through code analysis and testing. The AI analyzed: - ropey source code to understand the panic condition - Existing usage of `clip_offset` in the codebase - Tree-sitter byte offset behavior with UTF-8 All code has been reviewed and tested by humans, and further validated by Gemini AI. ## Checklist - [x] Follows existing code style - [x] One PR does one thing (UTF-8 panic fix only) - [x] All manual tests pass - [x] Tested with real multi-byte UTF-8 content - [x] No breaking changes - [x] No performance regression |
||
|---|---|---|
| .. | ||
| hello_world | ||
| macros | ||
| story | ||
| ui | ||