-
-
Notifications
You must be signed in to change notification settings - Fork 3.1k
[mypyc] Add str.isalnum() primitive
#20852
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -654,3 +654,26 @@ bool CPyStr_IsSpace(PyObject *str) { | |||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||
| return true; | ||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| bool CPyStr_IsAlnum(PyObject *str) { | ||||||||||||||||||||||||||||||||||
| Py_ssize_t len = PyUnicode_GET_LENGTH(str); | ||||||||||||||||||||||||||||||||||
| if (len == 0) return false; | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| if (PyUnicode_IS_ASCII(str)) { | ||||||||||||||||||||||||||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could this be
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Switching from ASCII path to the 1 byte puts a very big dent on performance, I assume because
However, I can combine all 4 cases (ASCII plus 3 byte kinds) and hide their for loops behind a macro
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Judging from these numbers, it seems like checking any non-trivial string with |
||||||||||||||||||||||||||||||||||
| const Py_UCS1 *data = PyUnicode_1BYTE_DATA(str); | ||||||||||||||||||||||||||||||||||
| for (Py_ssize_t i = 0; i < len; i++) { | ||||||||||||||||||||||||||||||||||
| if (!Py_ISALNUM(data[i])) | ||||||||||||||||||||||||||||||||||
| return false; | ||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||
| return true; | ||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| int kind = PyUnicode_KIND(str); | ||||||||||||||||||||||||||||||||||
| const void *data = PyUnicode_DATA(str); | ||||||||||||||||||||||||||||||||||
| for (Py_ssize_t i = 0; i < len; i++) { | ||||||||||||||||||||||||||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Performance might increase if there was a separate loop for 2 byte and 4 byte kinds. This way the read operation wouldn't need to branch based on kind, which might result in better code. Can you try this out?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I tried it locally, it only slightly reduced the tail end (still 13+ seconds for the 2 byte 100 length one) so we'd still spot a significant regression for the larger strings. I also tried calling What is the preferred action here, do we still keep the primitive in the hopes that most strings are small or should mypyc always be at least on par or better than CPython?
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This still looks better than CPython on average, as ASCII strings and short strings are common. To match CPython performance we might need to have a custom implementation of Py_UNICODE_ISALNUM, which doesn't seem worth it. I'll experiment with this a little, but this might be close to as good as we can easily achieve.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sounds good! I also wondered what it'd take to mirror |
||||||||||||||||||||||||||||||||||
| Py_UCS4 ch = PyUnicode_READ(kind, data, i); | ||||||||||||||||||||||||||||||||||
| if (!Py_UNICODE_ISALNUM(ch)) | ||||||||||||||||||||||||||||||||||
| return false; | ||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||
| return true; | ||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was not documented in the
str.isspace()PR, added it now