## Problem
SyntaxHighlighter panics when highlighting text containing multi-byte
UTF-8 characters (Chinese, Japanese, emoji, etc).
**Error**: `thread 'main' panicked at
ropey-2.0.0-beta.1/src/rope.rs:694:59`
**Root Cause**: tree-sitter may return byte offsets that fall in the
middle of multi-byte UTF-8 characters. When `Rope::slice()` receives
such offsets, it checks `is_char_boundary()` and panics with
`NonCharBoundary` error.
## Solution
Use `RopeExt::clip_offset()` to adjust byte offsets to the nearest char
boundaries before calling `slice()`.
**Changes**:
- Line 395-398: Clip offsets in main highlighting loop
- Line 441-443: Clip offsets when extracting injection content
- Line 478-479: Added explanatory comment for injection highlighting
- Added imports for `RopeExt` and `Bias`
**Safety**:
- `Bias::Left` for start offset ensures we don't skip the beginning of a
character
- `Bias::Right` for end offset ensures we include the full character
- The adjustment is minimal (at most 3 bytes for UTF-8)
- Uses existing project API (`RopeExt::clip_offset`)
## Testing
Tested with:
- ✅ Chinese text: "你好世界"、"**加粗中文**"
- ✅ Japanese text: "こんにちは"、"日本語"
- ✅ Emoji: "😀🎉✨🚀"
- ✅ Markdown syntax highlighting with CJK characters
- ✅ Code blocks with mixed languages
All tests pass without panics. Syntax highlighting works correctly. No
performance degradation observed.
## AI Assistance
🤖 This fix was developed with AI assistance (Claude). The solution
approach (using `clip_offset`) was identified through code analysis and
testing. The AI analyzed:
- ropey source code to understand the panic condition
- Existing usage of `clip_offset` in the codebase
- Tree-sitter byte offset behavior with UTF-8
All code has been reviewed and tested by humans, and further validated
by Gemini AI.
## Checklist
- [x] Follows existing code style
- [x] One PR does one thing (UTF-8 panic fix only)
- [x] All manual tests pass
- [x] Tested with real multi-byte UTF-8 content
- [x] No breaking changes
- [x] No performance regression
All WEF (Web Embedding Framework) related code has been successfully
removed from the repository as it has been extracted to
https://github.com/longbridge/wef
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: huacnlee <5518+huacnlee@users.noreply.github.com>
This change aim to fix the before version `shape_text` (in Element
paint) and `wrap_line` (in TextWrapper cached soft wrap map) may have
difference wrap width.
Now move to based on TextWrapper's wrap map to paint text by use
`shape_line` method.
