dan/rush
1
0
Fork 0
mirror of https://github.com/danbulant/rush synced 2026-05-24 12:35:38 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

preserve token positions

Browse source
This commit is contained in:
Daniel Bulant 2022-02-21 22:12:42 +01:00
parent 33571b0c03
commit b000e924e8
4 changed files with 164 additions and 161 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

86
README.md
View file

@ -4,27 +4,37 @@ Rust shell. Inspired by Ion.
In case you're reading this: rush is in the works and not a priority. Features may be missing even if defined below. In case you're reading this: rush is in the works and not a priority. Features may be missing even if defined below.
## Scopes
Variables are block scoped.
Block scope creation:
- `if`
- `while`
- `else`
- `for..`
- `$(expr)`
Functions have a copy of their scope.
Files create file scopes, to which functions are scoped.
## Syntax ## Syntax
`;` is 'alias' for new line. `;` is 'alias' for new line.
Syntax and type errors crash the program. Syntax and type errors crash the program.
Scopes are for each block. Functions won't have access to variables it wouldn't have access if it wasn't a function: Variables are scoped to their block, and immediately freed when their block is left.
```sh
fn testing
echo $t
end
if true
testing # Error! t is not defined
end
```
### Variables ### Variables
String variables using `$`, arrays using `@`. String variable value can be obtained using `$`, arrays using `@`.
When an array is stringified (referred to with `$`), it's contents are joined with space.
No special treatment of `PATH`.
Currently, the shell doesn't error out when variable doesn't exist, instead, it's replaced by an empty string.
Assigned using `let`. Assigned using `let`.
Left side is evaluated to a string as well. Left side is evaluated to a string as well.
@ -37,8 +47,9 @@ echo $d # c
``` ```
Arrays are assigned using `[ var ]`. You can join arrays and strings by simply passing them there, like `[ $var @var ]`. Arrays are assigned using `[ var ]`. You can join arrays and strings by simply passing them there, like `[ $var @var ]`.
Arrays and maps cannot be nested during definition (`[ $var [@var] ]` should have the same effect).
All assignments are done via the `let` keyword. All assignments are done via the `let` keyword. If the variable exists, it is overwritten (even in upper scopes).
Instead of `=`, other operations are supported: Instead of `=`, other operations are supported:
* `*=` - multiply * `*=` - multiply
@ -55,29 +66,9 @@ Instead of `=`, other operations are supported:
`env::` namespace contains the environment (and doesn't error out if the variable doesn't exist, instead, empty string is returned) `env::` namespace contains the environment (and doesn't error out if the variable doesn't exist, instead, empty string is returned)
`color::` (alias `c::`) has a number of colors `color::` (alias `c::`) has a number of colors
#### Types
Based on the value set in the set `let` (the one with just `=`), a type is infered (unless specificaly set using `let x:type = ...`). This type is then used for the operations after.
Supported types:
* `i32` (alias int)
* `i64`
* `i128`
* `u32`
* `u64`
* `u128`
* `f32`
* `f64`
* `str`
* `hmap[T]` (where T is one of the other types, except array)
* `[T]` (where T is one of the other types, except hmap)
HashMap is basically array, but with string keys (instead of numbers) in random order.
### Return ### Return
Sets the exit code (and possibly exits function/script early). If no return is set, the return code is set to the return code of the last expression. Sets the exit code (and possibly exits function/script early). If no return is set, the return code is set to the return code of the last expression (`$?`).
### Math ### Math
@ -99,29 +90,32 @@ Slices: `[x..y]` gets a substring (or subarray) of the variable. When `x` ommite
Bracketless. Scopes are ended by the keyword `end`. Bracketless. Scopes are ended by the keyword `end`.
`if` - Runs it's scope if the command returns `0`. Useful in pair with `test` builtin. - `if` - Runs it's scope if the command returns `0`. Useful in pair with `test` builtin. `else` supported. `else if` doesn't require another `end`.
`for $val of @arr` - Runs for each value of the array (or hashmap) - `for $val of @arr` - Runs for each value of the array (or hashmap)
`for $val of ...` - Runs for each number in the range. - `for $val of X..Y` - Runs for each number in the range `X` and `Y` (both inclusive).
`while` - Runs in loop as long as the command returns `0` - `while` - Runs in loop as long as the command returns `0`
### Functions ### Functions
Defined by `fn name arg -- desc`. `arg` can be ommited, or repeated. `desc` will be printed when the `arg` is missing (or when `describe` command is used). Defined by `fn name [...arg] [--flags]`. `arg` can be ommited, or repeated.
`--flags` can be used to add additional functionality.
#### Special Functions are scoped per file, even if they use `on-event` or similar to be triggered.
Use `source` to load external files with functions to be triggered.
From config (defined by `~/.rushrc`), special functions can be defined. - `--desc` sets the functions description.
- `--on-event` will run the function when an event is run
* `PROMPT` will be run to render the prompt
* `HIGHLIGHT` will run (for each key - make it fast) to highlight the text.
#### Builtins #### Builtins
* `let` for assigning variables `let` is a special case which cannot be dynamically addressed (i.e. using `$(echo let) var = value`).
* `export` for exporting variables to env
* `let` for assigning variables (`let var = value`)
* `export` for exporting variables to env (`export var` to export var, or `export var = value`)
* `test` tests for evaluation (`=` for equality, `>`, `<`, `<=`, `>=` for number comparisons) * `test` tests for evaluation (`=` for equality, `>`, `<`, `<=`, `>=` for number comparisons)
* `exists` for existance of a given string, or if given a flag (`-F`unctions, `-v`ariables, `-e`nv, `-f`ile, `-d`irectory, `-r`eadable file, `-w`ritable file, e`-x`ecutable file), existence of the selected object * `exists` for existance of a given string, or if given a flag (`-F`unctions, `-v`ariables, `-e`nv, `-f`ile, `-d`irectory, `-r`eadable file, `-w`ritable file, e`-x`ecutable file), existence of the selected object
* `true` returns `0` * `true` returns `0`
* `false` returns `1` * `false` returns `1`
* `source` to run another file in the same file scope
Some GNU standard utils may be overwritten by rush builtins, but must be made compatible. Some GNU standard utils may be overwritten by rush builtins, but must be made compatible.

150
src/main.rs
View file

@ -71,19 +71,82 @@ impl Shell {
let v = stdin.lock().lines().next().unwrap().unwrap(); let v = stdin.lock().lines().next().unwrap().unwrap();
self.term.input = v; self.term.input = v;
} }
}
fn start_shell() {
let mut shell = Shell::new(); fn edit(&mut self) {
loop { let stdin = io::stdin();
print!("$: "); let mut stdout = io::stdout().into_raw_mode().unwrap();
io::stdout().flush().unwrap(); for c in stdin.keys() {
shell.collect(); let c = c.unwrap();
shell.term.input += "\n"; match c {
let res = parser::exec(&mut shell.term.input.as_bytes(), &mut shell.ctx); Key::Char('\n') => {
match res { if self.term.input.chars().nth(self.term.idx).unwrap_or(' ') == '\\' {
Err(err) => eprintln!("rush: {}", err), self.term.insert_str(self.term.idx, "\\\n");
Ok(_) => {} } else {
break;
}
}
Key::Backspace => {
if self.term.input.len() > 0 && self.term.idx > 0 {
if self.term.idx == self.term.input.len() - 1 {
self.term.input.pop();
} else {
self.term.remove(self.term.idx - 1);
}
self.term.idx -= 1;
}
}
Key::Delete => {
if self.term.idx < self.term.input.len() {
self.term.remove(self.term.idx);
}
}
Key::End => {
self.term.idx = cmp::max(self.term.input.len(), 1) - 1;
}
Key::Home => {
self.term.idx = 0;
}
Key::Left => {
if self.term.idx > 0 {
self.term.idx -= 1;
}
}
Key::Right => {
if self.term.idx < self.term.input.len() - 1 {
self.term.idx += 1;
}
}
Key::Ctrl('c') => {
process::exit(1);
}
Key::Ctrl('d') => {
process::exit(0);
}
Key::Char(char) => {
self.term.insert(self.term.idx, char);
self.term.idx += 1;
}
_ => {}
}
self.term.print(&mut stdout);
stdout.flush().unwrap();
}
stdout.suspend_raw_mode().unwrap();
}
fn start() {
let mut shell = Shell::new();
loop {
print!("$: ");
io::stdout().flush().unwrap();
shell.collect();
shell.term.input += "\n";
let res = parser::exec(&mut shell.term.input.as_bytes(), &mut shell.ctx);
match res {
Err(err) => eprintln!("rush: {}", err),
Ok(_) => {}
}
} }
} }
} }
@ -127,7 +190,7 @@ fn main() {
}, },
None => {} None => {}
}; };
start_shell(); Shell::start();
} }
#[cfg(test)] #[cfg(test)]
@ -169,64 +232,3 @@ mod test {
load_and_run("test/while.rush") load_and_run("test/while.rush")
} }
} }
fn editor() -> Shell {
let stdin = io::stdin();
let mut stdout = io::stdout().into_raw_mode().unwrap();
let mut shell = Shell::new();
for c in stdin.keys() {
let c = c.unwrap();
match c {
Key::Char('\n') => {
if shell.term.input.chars().nth(shell.term.idx).unwrap_or(' ') == '\\' {
shell.term.insert_str(shell.term.idx, "\\\n");
} else {
break;
}
}
Key::Backspace => {
if shell.term.input.len() > 0 && shell.term.idx > 0 {
if shell.term.idx == shell.term.input.len() - 1 {
shell.term.input.pop();
} else {
shell.term.remove(shell.term.idx - 1);
}
shell.term.idx -= 1;
}
}
Key::Delete => {
if shell.term.idx < shell.term.input.len() {
shell.term.remove(shell.term.idx);
}
}
Key::End => {
shell.term.idx = cmp::max(shell.term.input.len(), 1) - 1;
}
Key::Home => {
shell.term.idx = 0;
}
Key::Left => {
if shell.term.idx > 0 {
shell.term.idx -= 1;
}
}
Key::Right => {
if shell.term.idx < shell.term.input.len() - 1 {
shell.term.idx += 1;
}
}
Key::Ctrl('c') => {
process::exit(1);
}
Key::Char(char) => {
shell.term.insert(shell.term.idx, char);
shell.term.idx += 1;
}
_ => {}
}
shell.term.print(&mut stdout);
stdout.flush().unwrap();
}
stdout.suspend_raw_mode().unwrap();
shell
}

22
src/parser/ast.rs
View file

@ -1,4 +1,4 @@
use crate::parser::tokens::Tokens; use crate::parser::tokens::{Token, Tokens};
use anyhow::{bail, Context, Result}; use anyhow::{bail, Context, Result};
#[derive(Debug)] #[derive(Debug)]
@ -126,7 +126,7 @@ pub enum Expression {
#[derive(Debug)] #[derive(Debug)]
struct Tree { struct Tree {
tokens: Vec<Tokens>, tokens: Vec<Token>,
i: usize i: usize
} }
@ -177,7 +177,7 @@ impl Tree {
buf.push(val); buf.push(val);
if self.i >= end - 1 { break } if self.i >= end - 1 { break }
self.i += 1; self.i += 1;
token = self.tokens.get(self.i).unwrap(); token = &self.tokens.get(self.i).unwrap().token;
if matches!(token, Tokens::CommandEnd(_)) { break } if matches!(token, Tokens::CommandEnd(_)) { break }
} }
match &token { match &token {
@ -196,7 +196,7 @@ impl Tree {
self.inc(); self.inc();
let mut len = 0; let mut len = 0;
for token in &self.tokens[self.i..] { for token in &self.tokens[self.i..] {
match token { match token.token {
Tokens::ExportSet => { break }, Tokens::ExportSet => { break },
_ => len += 1 _ => len += 1
} }
@ -218,7 +218,7 @@ impl Tree {
let mut found_first = false; let mut found_first = false;
for token in &self.tokens[self.i..] { for token in &self.tokens[self.i..] {
val_end += 1; val_end += 1;
match token { match token.token {
Tokens::Space => if found_first { break }, Tokens::Space => if found_first { break },
Tokens::CommandEnd(_) => if !found_first { bail!("Unexpected command end") } else { break }, Tokens::CommandEnd(_) => if !found_first { bail!("Unexpected command end") } else { break },
Tokens::FileRead => bail!("Unexpected file read (<)"), Tokens::FileRead => bail!("Unexpected file read (<)"),
@ -242,7 +242,7 @@ impl Tree {
let mut found_first = false; let mut found_first = false;
for token in &self.tokens[self.i..] { for token in &self.tokens[self.i..] {
val_end += 1; val_end += 1;
match token { match token.token {
Tokens::Space => if found_first { break }, Tokens::Space => if found_first { break },
Tokens::CommandEnd(_) => if !found_first { bail!("Unexpected command end") } else { break }, Tokens::CommandEnd(_) => if !found_first { bail!("Unexpected command end") } else { break },
Tokens::FileRead => bail!("Unexpected file read (<)"), Tokens::FileRead => bail!("Unexpected file read (<)"),
@ -381,7 +381,7 @@ impl Tree {
let mut lvl = 1; let mut lvl = 1;
self.inc(); self.inc();
for token in &self.tokens[self.i..] { for token in &self.tokens[self.i..] {
match token { match token.token {
Tokens::SubStart => lvl += 1, Tokens::SubStart => lvl += 1,
Tokens::StringFunction(_) => lvl += 1, Tokens::StringFunction(_) => lvl += 1,
Tokens::ArrayFunction(_) => lvl += 1, Tokens::ArrayFunction(_) => lvl += 1,
@ -468,7 +468,7 @@ impl Tree {
let mut lvl = 1; let mut lvl = 1;
self.inc(); self.inc();
for token in &self.tokens[self.i..] { for token in &self.tokens[self.i..] {
match token { match token.token {
Tokens::ParenthesisStart => lvl += 1, Tokens::ParenthesisStart => lvl += 1,
Tokens::ParenthesisEnd => lvl -= 1, Tokens::ParenthesisEnd => lvl -= 1,
_ => {} _ => {}
@ -547,11 +547,11 @@ impl Tree {
self.i += 1; self.i += 1;
self self
} }
fn get_current_token(&self) -> &Tokens { self.tokens.get(self.i).unwrap() } fn get_current_token(&self) -> &Tokens { &self.tokens.get(self.i).unwrap().token }
fn get_next_token(&self) -> &Tokens { self.tokens.get(self.i + 1).unwrap() } fn get_next_token(&self) -> &Tokens { &self.tokens.get(self.i + 1).unwrap().token }
} }
pub fn build_tree(tokens: Vec<Tokens>) -> Result<Vec<Expression>> { pub fn build_tree(tokens: Vec<Token>) -> Result<Vec<Expression>> {
let mut expressions: Vec<Expression> = Vec::new(); let mut expressions: Vec<Expression> = Vec::new();
let mut tree = Tree { tokens, i: 0 }; let mut tree = Tree { tokens, i: 0 };
loop { loop {

67
src/parser/tokens.rs
View file

@ -1,5 +1,12 @@
use anyhow::{Result, bail}; use anyhow::{Result, bail};
#[derive(Debug)]
pub struct Token {
pub token: Tokens,
pub start: usize,
pub end: usize
}
#[derive(Debug)] #[derive(Debug)]
pub enum Tokens { pub enum Tokens {
Space, Space,
@ -86,7 +93,7 @@ impl Tokens {
} }
fn read_var_ahead(i: usize, text: &String) -> (usize, Tokens) { fn read_var_ahead(i: usize, text: &String) -> (usize, Token) {
let mut x = i; let mut x = i;
let mut buf = String::new(); let mut buf = String::new();
let parens_mode = text.chars().nth(x + 1).unwrap() == '{'; let parens_mode = text.chars().nth(x + 1).unwrap() == '{';
@ -113,14 +120,14 @@ fn read_var_ahead(i: usize, text: &String) -> (usize, Tokens) {
} }
} }
let token = match text.chars().nth(i).unwrap() { let token = match text.chars().nth(i).unwrap() {
'$' => Tokens::StringVariable(buf, parens_mode), '$' => Token { token: Tokens::StringVariable(buf, parens_mode), start: i, end: i + x },
'@' => Tokens::ArrayVariable(buf, parens_mode), '@' => Token { token: Tokens::ArrayVariable(buf, parens_mode), start:i , end: i+x },
a => panic!("Invalid value {}", a) a => panic!("Invalid value {}", a)
}; };
(x - i - 1, token) (x - i - 1, token)
} }
pub fn tokenize(reader: &mut dyn std::io::BufRead) -> Result<Vec<Tokens>> { pub fn tokenize(reader: &mut dyn std::io::BufRead) -> Result<Vec<Token>> {
let mut quote_active = false; let mut quote_active = false;
let mut double_quote_active = false; let mut double_quote_active = false;
let mut escape_active = false; let mut escape_active = false;
@ -128,10 +135,10 @@ pub fn tokenize(reader: &mut dyn std::io::BufRead) -> Result<Vec<Tokens>> {
reader.read_to_string(&mut text)?; reader.read_to_string(&mut text)?;
let mut text_length = text.len(); let mut text_length = text.len();
let mut tokens: Vec<Tokens> = Vec::new(); let mut tokens: Vec<Token> = Vec::new();
fn save_buf(buf: &mut String, tokens: &mut Vec<Tokens>) { fn save_buf(buf: &mut String, tokens: &mut Vec<Token>, i: usize) {
if buf.len() > 0 { tokens.push(Tokens::detect(std::mem::take(buf))) } if buf.len() > 0 { tokens.push(Token { token: Tokens::detect(std::mem::take(buf)), end: i, start: i - buf.len() }) }
} }
let mut buf = String::new(); let mut buf = String::new();
@ -147,22 +154,22 @@ pub fn tokenize(reader: &mut dyn std::io::BufRead) -> Result<Vec<Tokens>> {
'"' => if !escape_active && !quote_active { double_quote_active = !double_quote_active; buf_add = false }, '"' => if !escape_active && !quote_active { double_quote_active = !double_quote_active; buf_add = false },
'\'' => if !escape_active && !double_quote_active { quote_active = !quote_active; buf_add = false }, '\'' => if !escape_active && !double_quote_active { quote_active = !quote_active; buf_add = false },
'$' => if !escape_active && !quote_active { '$' => if !escape_active && !quote_active {
save_buf(&mut buf, &mut tokens); save_buf(&mut buf, &mut tokens, i);
if text_length > i && text.chars().nth(i + 1).unwrap() == '(' { if text_length > i && text.chars().nth(i + 1).unwrap() == '(' {
tokens.push(Tokens::SubStart); tokens.push(Token { token: Tokens::SubStart, start: i, end: i+1 });
skipper = 1; skipper = 1;
buf_add = false; buf_add = false;
} else { } else {
let (skippers, mut token) = read_var_ahead(i, &text); let (skippers, mut token) = read_var_ahead(i, &text);
match token { match token.token {
Tokens::StringVariable(ref str, bool) => if !bool && !double_quote_active { Tokens::StringVariable(ref str, bool) => if !bool && !double_quote_active {
if text.len() > i + skippers && text.chars().nth(i + skippers).unwrap() == '(' { if text.len() > i + skippers && text.chars().nth(i + skippers).unwrap() == '(' {
token = Tokens::StringFunction(str.clone()); token = Token { token: Tokens::StringFunction(str.clone()), end: i + skippers, start: i };
} }
}, },
Tokens::ArrayVariable(ref str, bool) => if !bool && !double_quote_active { Tokens::ArrayVariable(ref str, bool) => if !bool && !double_quote_active {
if text.len() > i + skippers && text.chars().nth(i + skippers).unwrap() == '(' { if text.len() > i + skippers && text.chars().nth(i + skippers).unwrap() == '(' {
token = Tokens::ArrayFunction(str.clone()); token = Token { token: Tokens::ArrayFunction(str.clone()), end: i+skippers, start: i };
} }
} }
_ => bail!("Cannot happen") _ => bail!("Cannot happen")
@ -173,8 +180,8 @@ pub fn tokenize(reader: &mut dyn std::io::BufRead) -> Result<Vec<Tokens>> {
} }
}, },
';' | '\r' | '\n' => if !escape_active && !quote_active && !double_quote_active { ';' | '\r' | '\n' => if !escape_active && !quote_active && !double_quote_active {
save_buf(&mut buf, &mut tokens); save_buf(&mut buf, &mut tokens, i);
tokens.push(Tokens::CommandEnd(letter.clone())); tokens.push(Token { token: Tokens::CommandEnd(letter.clone()), start: i, end: i });
let mut x = 0; let mut x = 0;
while x < text.len() - 1 && matches!(text.chars().nth(x).unwrap(), '\n' | '\r' | ';' | ' ') { while x < text.len() - 1 && matches!(text.chars().nth(x).unwrap(), '\n' | '\r' | ';' | ' ') {
x += 1; x += 1;
@ -185,28 +192,28 @@ pub fn tokenize(reader: &mut dyn std::io::BufRead) -> Result<Vec<Tokens>> {
buf_add = false; buf_add = false;
}, },
'&' => if !escape_active && !quote_active && !double_quote_active { '&' => if !escape_active && !quote_active && !double_quote_active {
save_buf(&mut buf, &mut tokens); save_buf(&mut buf, &mut tokens, i);
if i + 1 < text.len() && text.chars().nth(i+1).unwrap() == '&' { if i + 1 < text.len() && text.chars().nth(i+1).unwrap() == '&' {
tokens.push(Tokens::And); tokens.push(Token { token: Tokens::And, start: i, end: i+1 });
skipper = 1; skipper = 1;
} else { } else {
tokens.push(Tokens::JobCommandEnd); tokens.push(Token { token: Tokens::JobCommandEnd, start: i , end: i });
} }
buf_add = false; buf_add = false;
}, },
'|' => if !escape_active && !quote_active && !double_quote_active { '|' => if !escape_active && !quote_active && !double_quote_active {
save_buf(&mut buf, &mut tokens); save_buf(&mut buf, &mut tokens, i);
if i + 1 < text.len() && text.chars().nth(i+1).unwrap() == '|' { if i + 1 < text.len() && text.chars().nth(i+1).unwrap() == '|' {
tokens.push(Tokens::Or); tokens.push(Token { token: Tokens::Or, start: i, end: i+1 });
skipper = 1; skipper = 1;
} else { } else {
tokens.push(Tokens::RedirectInto); tokens.push(Token { token: Tokens::RedirectInto, start: i, end: i });
} }
buf_add = false; buf_add = false;
}, },
' ' => if !escape_active && !quote_active && !double_quote_active { ' ' => if !escape_active && !quote_active && !double_quote_active {
save_buf(&mut buf, &mut tokens); save_buf(&mut buf, &mut tokens, i);
tokens.push(Tokens::Space); tokens.push(Token { token: Tokens::Space, start: i, end: i });
let mut x = i; let mut x = i;
while text.chars().nth(x).unwrap() == ' ' { while text.chars().nth(x).unwrap() == ' ' {
x += 1; x += 1;
@ -215,13 +222,13 @@ pub fn tokenize(reader: &mut dyn std::io::BufRead) -> Result<Vec<Tokens>> {
buf_add = false; buf_add = false;
}, },
'(' => if !quote_active && !double_quote_active && !escape_active { '(' => if !quote_active && !double_quote_active && !escape_active {
save_buf(&mut buf, &mut tokens); save_buf(&mut buf, &mut tokens, i);
tokens.push(Tokens::ParenthesisStart); tokens.push(Token { token: Tokens::ParenthesisStart, start: i, end: i });
buf_add = false; buf_add = false;
} }
')' => if !quote_active && !double_quote_active && !escape_active { ')' => if !quote_active && !double_quote_active && !escape_active {
save_buf(&mut buf, &mut tokens); save_buf(&mut buf, &mut tokens, i);
tokens.push(Tokens::ParenthesisEnd); tokens.push(Token { token: Tokens::ParenthesisEnd, start: i, end: i });
buf_add = false; buf_add = false;
}, },
'\\' => if !escape_active { '\\' => if !escape_active {
@ -231,12 +238,12 @@ pub fn tokenize(reader: &mut dyn std::io::BufRead) -> Result<Vec<Tokens>> {
escape_active = false; escape_active = false;
}, },
'=' => if !escape_active && !quote_active && !double_quote_active { '=' => if !escape_active && !quote_active && !double_quote_active {
save_buf(&mut buf, &mut tokens); save_buf(&mut buf, &mut tokens, i);
tokens.push(Tokens::ExportSet); tokens.push(Token { token: Tokens::ExportSet, start: i, end: i });
buf_add = false; buf_add = false;
}, },
'#' => if !escape_active && !quote_active && !double_quote_active { '#' => if !escape_active && !quote_active && !double_quote_active {
save_buf(&mut buf, &mut tokens); save_buf(&mut buf, &mut tokens, i);
buf_add = false; buf_add = false;
let mut x = 0; let mut x = 0;
while x + i + 1 < text.len() && text.chars().nth(x + i + 1).unwrap() != '\n' { while x + i + 1 < text.len() && text.chars().nth(x + i + 1).unwrap() != '\n' {
@ -251,7 +258,7 @@ pub fn tokenize(reader: &mut dyn std::io::BufRead) -> Result<Vec<Tokens>> {
buf.push(*letter); buf.push(*letter);
} }
} }
save_buf(&mut buf, &mut tokens); save_buf(&mut buf, &mut tokens, text.len());
Ok(tokens) Ok(tokens)
} }