Getting the attributes of integers and floats
python
Solution
It's probably a consequence of the parsing algorithm used. A simple mental model is that the tokenizer attempts to match all the token patterns there are, and recognizes the longest match it finds. On a lower-level, the tokenizer works character-by-character, and makes a decision based only on the current state and input character – there shouldn't be any backtracking or re-reading of input.
After joining patterns with common prefixes – in this case, the pattern for `int` literals and the integral part of the pattern of `float` literals – what happens in the tokenizer is that it:
- Reads the `1`, and enters the state that indicates "reading either a `float` or an `int` literal"
- Reads the `.`, and enters the state "reading a `float` literal"
- Reads the `_`, which can not be part of a `float` literal. The parser emits `1.` as a `float` literal token.
- Carries on parsing starting with the `_`, and eventually emits `__class__` as an identifier token.
Aside: This tokenizing approach is also the reason why common languages have the syntax restrictions they have. E.g. identifiers contain letters, digits, and underscores, but cannot start with a digit. If that was allowed, `123abc` could be intended as either an identifier, or the integer `123` followed by the identifier `abc`.
A lex-like tokenizer would recognize this as the former since it leads to the longest single token, but nobody likes having to keep details like this in their head when trying to read code. Or when trying to write and debug the tokenizer for that matter.
The parser then tries to process the token stream:
<FloatLiteral: '1.'> <Identifier: '__class__'>
In Python, a literal directly followed by an identifier – without an operator between the tokens – makes no sense, so the parser bails. This also means that the reason why Python would complain about `123abc` being invalid syntax isn't the tokenizer error "the character `a` isn't valid in an integer literal", but the parser error "the identifier `abc` cannot directly follow the integer literal `123`"
The reason why the tokenizer can't recognize the `1` as an `int` literal is that the character that makes it leave the `float`-or-`int` state determines what it just read. If it's `.`, it was the start of a `float` literal, which might continue afterwards. If it's something else, it was a complete `int` literal token.
It's not possible for the tokenizer to "go back" and re-read the previous input as something else. In fact, the tokenizer is at too low a level to care about what an "attribute access" is and handle such ambiguities.
Now, your second example is valid because the tokenizer knows a `float` literal can only have one `.` in it. More precisely: the first `.` makes it transition from the `float`-or-`int` state to the `float` state. In this state, it only expects digits (or an `E` for scientific/engineering notation, a `j` for complex numbers…) to continue the the `float` literal. The first character that's not a digit etc. (i.e. the `.`) is definitely no longer part of the `float` literal and the tokenizer can emit the finished token. The token stream for your second example will thus be:
<FloatLiteral: '1.'> <Operator: '.'> <Identifier: '__class__'>
Which, of course, the parser then recognizes as valid Python. Now we also know enough why the suggested workarounds help. In Python, separating tokens with whitespace is optional – unlike, say, in Lisp. Conversely, whitespace does separate tokens. (That is, no tokens except `string` literals may contain whitespace, it's merely skipped between tokens.) So the code:
1 .__class__
is always tokenized as
<IntLiteral: '1'> <Operator: '.'> <Identifier: '__class__'>
And since a closing parenthesis cannot appear in an `int` literal, this:
(1).__class__
gets read as this:
<Operator: '('> <IntLiteral: '1'> <Operator: ')'> <Operator: '.'> <Identifier: '__class__'>
The above implies that, amusingly, the following is also valid:
1..__class__ # => <type 'float'>
The decimal part of a `float` literal is optional, and the second `.` read will make the preceding input be recognized as one.
Problem
I was playing with the Python interpreter (Python 3.2.3) and tried the following: ``` >>> dir(1) ``` This gave me all the attributes and methods of the int object. Next I tried: ``` >>> 1.__class__ ``` However this threw an exception: ``` File "<stdin>", line 1 1.__class__ ^ SyntaxError: invalid syntax ``` When I tried out the same with a float I got what I expected: ``` >>> 2.0.__class__ <class 'float'> ``` Why do `int` and `float` literals behave differently?