Flaky test_annotationlib test due to non-deterministic set iteration ordering

[colesbury]

Seen in a 3.14 buildbot

test_type_repr (test.test_annotationlib.TestTypeRepr.test_type_repr) ... ok

======================================================================
FAIL: test_nonexistent_attribute (test.test_annotationlib.TestForwardRefFormat.test_nonexistent_attribute)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/buildbot/buildarea/3.14.ware-debian-x86.installed/build/target/lib/python3.14/test/test_annotationlib.py", line 161, in test_nonexistent_attribute
    self.assertEqual(epsilon_anno, support.EqualToForwardRef("some | {obj, module}", owner=f))
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: ForwardRef('some | {module, obj}', owner=[79 chars]810>) != EqualToForwardRef('some | {obj, module}',[86 chars]810>)

----------------------------------------------------------------------
Ran 110 tests in 0.079s

FAILED (failures=1)

https://buildbot.python.org/#/builders/1793/builds/879/steps/8/logs/stdio

I stuck this into Claude and it the cause was due to non-deterministic set ordering, which I think is correct. I stuck a random.shuffle(elts) below and was able to repro the issue locally:

cpython/Lib/annotationlib.py

Lines 390 to 399 in a01694d

	elif type(other) in (list, tuple, set):
	extra_names = {}
	elts = []
	for elt in other:
	new_elt, new_extra_names = self.__convert_to_ast(elt)
	if new_extra_names is not None:
	extra_names.update(new_extra_names)
	elts.append(new_elt)
	ast_class = {list: ast.List, tuple: ast.Tuple, set: ast.Set}[type(other)]
	return ast_class(elts), extra_names

Looks related to:

• annotationlib: Improve test coverage #141174

cc @JelleZijlstra

Linked PRs

• gh-144348: annotationlib: fix test that relies on set ordering #144359

[Muawiya-contact]

Hi @colesbury, this flaky test makes sense due to the non-deterministic set iteration ordering. I’d like to help investigate and possibly fix the test to make it deterministic. Could you please assign me so I can start working on it?

[StanFromIreland]

@Muawiya-contact please stop posting the same request under multiple issues, and furthermore, Jelle has already assigned himself here.

[JelleZijlstra]

Thanks for reporting and sorry for the trouble. I'm sort of surprised this hasn't come up more, shouldn't this fail 50% of the time?

[Muawiya-contact]

@StanFromIreland Understood, thanks for the clarification. I’ll avoid cross-posting requests and won’t pursue assignment here since it’s already taken.