Vtyjkyuk

Example: the latex code:

documentclassarticle
begindocument

% test-case 1
% Output: "b"
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup

% test-case 2
% Output: "b"
% Expansion is allowed, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup

% test-case 3
% Compile error: cq is undefined
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup

% test-case 4
% Output: "B"
% Expansion is allowed, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup

enddocument

Here are my questions:

• 
  

  Whether the implicit token is allowed if expansion is allowed in the context?

  
  
  

  This might be the only way to explain the test-case 1, 2, 3.But it can not explain test-case 4.

  
  



• If not, when it is allowed?

I have read the TeXByTopic but I can't find a certain conclusion.

Implicit tokens are not expanded to their meaning, they are their meaning. So with

lettokenAlias=b

tokenAlias is an b. That shows up in your second test: tokenAlias and b are entirely equivalent.

However, case changing takes place based on the character code of the token concerned. (As noted in a comment, uppercase is also not expandable.) Thus tokenAlias is not affected by uppercase as it does not have a character code at all, being a control sequence.

The only way to get implicit character tokens to 'appear naturally' is to iterate over input and use a test on the meaning of each one, replacing every implicit token by it's meaning.

Taking the tests one at a time (following the edit), no expansion takes place in any cases. Thus:

1. implicitToken remains unchanged as it does not have a character code in the upper case table




2. implicitToken is b, so comparing the two using if is true: there is no expansion of implicitToken.




3. cq is never defined, and so the let and def lines have no effect




4. The edef does nothing as implicitToken is not expanded. Rather, we end up with cq with replacement text implicitToken: the effect this has thus depends on the meaning at point-of-use.

Note (again) that typesetting is not an expansion context: the result depends on the execution of the token.

Let's analyze your tests one by one.

An implicit token (more precisely an implicit character token) is a symbolic token defined by let<token>=<character>.

Test 2

% test-case 2
% Output: "b"
% Expansion is enabled, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup

This outputs “b”, because an implicit token behaves like the character it is defined to be equal to in the context of if or ifcat.

Test 3

% test-case 3
% Compile error: cq is undefined
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup

This is easier: the assignment defimplicitToken overrides the let. On the other hand, implicitToken is never used and cq (undefined) raises an error.

Test 4

% test-case 4
% Output: "B"
% Expansion is enabled, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup

If you add showcq after doing the edef you'll see

> cq=macro:
->implicitToken .

and the output will contain a “B”, because of the subsequent redefinition.

An implicit token is not expandable, so the codes

letimplicitToken=b
edefcqimplicitToken

and

letimplicitToken=b
defcqimplicitToken

are completely equivalent. TeX will use the meaning of implicitToken current at expansion of cq time: in the case of your test it is “B”.

Test 1

% test-case 1
% Output: "b"
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup

The primitive uppercase examines its argument (without macro expansion, but it's irrelevant here) and transforms each explicit character token into its uppercase form (as stored in the uccode vector). It doesn't act on implicit tokens.

Comments

There is an asymmetry: with letimplicitToken=b, the test ifimplicitToken b will return true. Similarly ifcat will use the category code of the character current at definition time; for instance

letimplicitToken=b
catcode`b=12
ifcatimplicitToken a%
 messageletter%
else
 messagenonletter%
fi

will print letter on the console, whereas ifcat ba would print nonletter.

Why this asymmetry? Sorry, but only Donald Knuth can answer. The most prominent usage of implicit tokens is with bgroup and egroup, which can be used as substitutes for and in certain (not all) contexts. I guess that making them subject to transformations in uppercase or lowercase would be irksome.

Expansion

Tokens in TeX can be expandable or unexpandable. A token of the latter kind, when found alone, is directly passed to the execution processor. What does this mean? Let's see an example. The primitive dimen is not expandable. If TeX finds dimen2=12pt, it will pass dimen to the execution processor; the execution prompts the search in the input stream for a <number>, an <optional equal> and a suitable value (with expansion). On the contrary, in advancedimen2 by 2pt, the token dimen is absorbed as the argument to advance for further processing. Non active (explicit) character tokens are not expandable; an active character can be or not, depending on how it's meaning is defined.

What tokens are expandable? Easy:

1. some primitives, such as conditionals, string, noexpand and others;


2. 
   macros, that is, tokens defined with def, edef, gdef or xdef;


3. tokens let equal to an expandable token.

Since your implicitToken belongs to none of the above types, it is unexpandable.