examples: adjust conformance server for auth tests

[maciej-kisiel]

This is a draft.

Early thoughts with regard to the interface:

• So far we started the server, here we need to instruct the conformance tests to start it, which is a change in approach and requires additional handling.
• The tests started working sooner than I thought they would, I need to analyze if that's expected.

Internal design notes:

• Not clear if extending the existing server is a good approach vs. creating a simpler auth-specific one.
  • Especially that the Go build flag that currently guards some of the libraries "poisoned" the conformance server now.
• The results of conformance tests do not land in the directory passed via a flag. To be investigated.

[jba]

I know this isn't ready for review, but could you provide a bit more context? Are there conformance tests that require auth? If so, we need this. But even if not, it still might be worth it as a good example for how to write a server that includes auth.

[maciej-kisiel]

Yes, such conformance tests are being added in modelcontextprotocol/conformance#105, which @pcarleton asked us on Discord to provide feedback for. I thought it might be a good exercise to assimilate the code base you explained to me last week. Sorry for not including the context in the first place.

[pcarleton]

Thanks @maciej-kisiel !

Not clear if extending the existing server is a good approach vs. creating a simpler auth-specific one.

My inclination here was to do a simpler separate server, and i'd love feedback on that decision. The main reason is that it would be annoying to have every non-auth related test pay the auth overhead, since presumably they would all return 401's if you don't send an access token. Splitting a separate server has downsides as well since then you basically can't run all server tests w/ 1 command.

Some other options in the running:

• keep auth on the everything server, but pass a dummy auth token that bypasses it (a little iffy)
• address the "single command / single report" issue by driving the conformance tests from a config file that can map suites & scenarios to different URL's or server starting commands
• stick the auth-requiring server on a sub-url of the main server (solves the single server process thing, but not much else)

The results of conformance tests do not land in the directory passed via a flag. To be investigated.

lmk if you find a bug here, this was merged recently. i made the default not outputting the report since i found I wasn't looking at it frequently. you may also be interested in conformance#99 which is a new feature we're hoping to land this week for passing in a baseline file to make it easier to ratchet tests up.