Lysandros Nikolaou d87b015106 gh-119118: Fix performance regression in tokenize module (#119615) ...

* gh-119118: Fix performance regression in tokenize module

- Cache line object to avoid creating a Unicode object
  for all of the tokens in the same line.
- Speed up byte offset to column offset conversion by using the
  smallest buffer possible to measure the difference.

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

2024-05-28 19:17:49 +00:00

11 KiB

Raw Blame History

View Raw