greg
|
7311d0311f
|
Simplify pattern a little bit
|
2017-09-19 22:10:58 -07:00 |
|
greg
|
1b59c264b4
|
Use itertools peeking_take_while
Cuts down on lines in the tokenizer
|
2017-09-19 22:10:58 -07:00 |
|
greg
|
b2e453a9de
|
Rewrite of tokenizer
|
2017-09-19 22:10:58 -07:00 |
|
greg
|
297003c0b0
|
Operator only needs to be a tuple struct
|
2017-09-19 22:10:58 -07:00 |
|
greg
|
e5ee072b00
|
Convert tokenizer to large match statement
In the hopes of making it shorter
|
2017-09-19 22:10:58 -07:00 |
|
greg
|
3063de1242
|
Run rustfmt on the rest of them
|
2017-09-19 22:10:57 -07:00 |
|
greg
|
743311d18a
|
Convert Tokenize to Result
|
2017-09-19 22:10:57 -07:00 |
|
greg
|
032d01c9f5
|
Fix tokenization bug
|
2017-09-19 22:10:55 -07:00 |
|
greg
|
c4ab1ed105
|
Fix tokenizer tests
|
2017-09-19 22:10:55 -07:00 |
|
greg
|
9a257f08d7
|
Introduce Op type
For operator parsing
|
2017-09-19 22:10:55 -07:00 |
|
greg
|
47d56a7b44
|
fix operator parsing
|
2017-09-19 22:10:55 -07:00 |
|
greg
|
e1aa7ecb17
|
Finish tokenizing Op separately
|
2017-09-19 22:10:55 -07:00 |
|
greg
|
f09a6e14ba
|
Tokenizer work to support operators
work in progress but committing to transfer
|
2017-09-19 22:10:55 -07:00 |
|
greg
|
bd1c455dc8
|
Basic infrastructure parses
Also got rid of EOF, don't need it
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
8c473c554e
|
Fix bug
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
3e04cbfa29
|
Add comma tokenization
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
570650cbfa
|
Finish keyword tokenization
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
49be163181
|
Add test to ignore
For better handing of user-defined operators, which I will do in the
future
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
b4f93acbd8
|
Couple more tests
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
8c65ae3214
|
Macro-ize token tests
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
e436533638
|
Passing test
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
4601a56867
|
Start working on tokenization tests
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
2f7a1850db
|
Finish tokenizing
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
71aef379d3
|
Tokenize number literals
TODO: expand this bit of code to handle 0x12, etc. syntax
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
8662a3ba0e
|
Make tokenize error-able
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
13cde3106c
|
Start making tokenizer changes
Hopefully this time iron out all the bugs from the last implementation
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
09d524c74a
|
Changing how parsing works again
Someone wrote a port of the LLVM kaleidoscope tutorial to rust, namely
https://github.com/jauhien/iron-kaleidoscope
I'm just gonna follow this along
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
dd2b4893a4
|
Get rid of Separator token
Have separate newline and semicolon tokens
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
509ab80b9c
|
I can now parse one thing
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
be98f8387e
|
Add another test
|
2017-09-19 22:10:54 -07:00 |
|
greg
|
841b38d5b1
|
Add tokenization test
|
2017-09-19 22:10:53 -07:00 |
|
greg
|
3af7e6a409
|
Refactoring Schala
Gonna work on Schala in earnest now! Using the simplerepl crate instead
of a build-in REPL, temporarily dropping parsing and evaluation code.
|
2017-09-19 22:10:53 -07:00 |
|
greg
|
bb349cda5f
|
Fix newline tokenizing bug
Still need to fix <statements> parsing because of the final newline
|
2017-09-19 22:10:53 -07:00 |
|
greg
|
e870d8172a
|
Make = a keyword
|
2017-09-19 22:10:53 -07:00 |
|
greg
|
4ef93fafb5
|
Cause tokenize error for unclosed strings
|
2017-09-19 22:10:53 -07:00 |
|
greg
|
9d6dc5a5f2
|
Tokenize periods separately
|
2017-09-19 22:10:52 -07:00 |
|
greg
|
0e4469fa58
|
Filled out keyword tokenizing
|
2017-09-19 22:10:52 -07:00 |
|
greg
|
edf100b583
|
Starting to do if statement parsing
|
2017-09-19 22:10:52 -07:00 |
|
greg
|
169e662049
|
Collapse Separator tokens
only ever gonna be one in a row
|
2017-09-19 22:10:52 -07:00 |
|
greg
|
bfa16fd6fb
|
Added Keyword lexical class
|
2017-09-19 22:10:52 -07:00 |
|
greg
|
1059a88ee6
|
Separate parsing into module
|
2017-09-19 22:10:52 -07:00 |
|
greg
|
429ace73bd
|
Move tokenizing into separate module
|
2017-09-19 22:10:52 -07:00 |