Skip to contentSkip to conversation

allow show_token() on TOKEN_ZERO_IDENT

PATCH In review Luc Van Oostenryck 3 participants 3 messages Jun 7, 2022 – Jun 8, 2022 Sparse checker

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 allow show_token() on TOKEN_ZERO_IDENT

From: Luc Van Oostenryck <[email protected]>

TOKEN_ZERO_IDENTs are created during the evaluation of pre-processor expressions but which otherwise are normal idents and were first tokenized as TOKEN_IDENTs.

As such, they could perfectly be displayed by show_token() but are not. So, in error messages they are displayed as "unhandled token type '4'", which is not at all informative.

Fix this by letting show_token() process them like usual TOKEN_IDENTs. Idem for quote_token().

---
 tokenize.c | 2 ++
 1 file changed, 2 insertions(+)
tokenize.c +2 -0
index ea7105438270..fdaea370cc48 100644
@@ -201,6 +201,7 @@ const char *show_token(const struct token *token)
 		return "end-of-input";
 
 	case TOKEN_IDENT:
+	case TOKEN_ZERO_IDENT:
 		return show_ident(token->ident);
 
 	case TOKEN_NUMBER:
@@ -259,6 +260,7 @@ const char *quote_token(const struct token *token)
 		return "syntax error";
 
 	case TOKEN_IDENT:
+	case TOKEN_ZERO_IDENT:
 		return show_ident(token->ident);
 
 	case TOKEN_NUMBER:
Signature · 1 line
2.36.1
#1 lore raw
Quoted 11 lines · Luc Van Oostenryck

TOKEN_ZERO_IDENTs are created during the evaluation of pre-processor expressions but which otherwise are normal idents and were first tokenized as TOKEN_IDENTs.

As such, they could perfectly be displayed by show_token() but are not. So, in error messages they are displayed as "unhandled token type '4'", which is not at all informative.

Fix this by letting show_token() process them like usual TOKEN_IDENTs. Idem for quote_token().

Ack.

I do wonder if it should be marked somehow as being that special case. The main reason for 'show_token()' is debugging, after all, and TOKEN_ZERO_IDENT does have magical properties in how it either silently expands to the constant '0', or it generates a warning about undefined preprocessor symbol.

But considering that we've apparently reported it as "unhandled token type '4'" since 2005, I guess it's not exactly a big deal.

Linus

#2 lore raw
Quoted 15 lines · Linus Torvalds
Quoted 11 lines · Luc Van Oostenryck

TOKEN_ZERO_IDENTs are created during the evaluation of pre-processor expressions but which otherwise are normal idents and were first tokenized as TOKEN_IDENTs.

As such, they could perfectly be displayed by show_token() but are not. So, in error messages they are displayed as "unhandled token type '4'", which is not at all informative.

Fix this by letting show_token() process them like usual TOKEN_IDENTs. Idem for quote_token().

Ack.

Quoted 8 lines

I do wonder if it should be marked somehow as being that special case. The main reason for 'show_token()' is debugging, after all, and TOKEN_ZERO_IDENT does have magical properties in how it either silently expands to the constant '0', or it generates a warning about undefined preprocessor symbol.

But considering that we've apparently reported it as "unhandled token type '4'" since 2005, I guess it's not exactly a big deal.

Yes, I first thought to do so but then choose not because I could not convince myself that its special property was irrelevant in warning/error messages. It looks to me more as an internal thing, more semantical than lexical, and a non-faithful representation would be confusing in messages.

For context, the input text I had (from GCC's testsuite) was:

	#define empty
	#if empty#cpu(m68k)
	#endif

and the error message sparse issued was:
error: garbage at end: #unhandled token type '4' (unhandled token type '4' )
with this patch it's:
error: garbage at end: #cpu(m68k)

-- Luc