Fix handling of WOLFSSH_USERAUTH_REJECTED result from userAuthCb

[tjko]

I noticed that if compile option NO_FAILURE_ON_REJECTED is not used,
then WOLFSSH_USERAUTH_REJECTED response from user authentication
callback (userAuthCb) do not appear to be handled properly (?) As this seems to prevent SSH server
(userAuthCb) from rejecting client connections.

If NO_FAILURE_ON_REJECTED is defined, then everything works ok, as "DoUserAuth" functions (DoUserAuthInforResponse(), DouserAuthRequestPassword(), DoUserAuthRequestPublicKey())
end up returning correct return value (WS_USER_AUTH_E).

But when NO_FAILURE_ON_REJECT is not defined, that causes authFailure getting set:

            else if (ret == WOLFSSH_USERAUTH_REJECTED) {
                WLOG(WS_LOG_DEBUG, "DUARKB: keyboard rejected");
                #ifndef NO_FAILURE_ON_REJECTED
                    authFailure = 1;
                #endif
                ret = WS_USER_AUTH_E;
            }

Which in turn results in the return value getting overridden little bit later by function call to SendUserAuthFailure():

    if (authFailure || partialSuccess) {
        ret = SendUserAuthFailure(ssh, partialSuccess);
    }

I noticed that DoUserAuthRequestNone() doesn't seem to have this problem as it handles this differently, so I updated the other "DoUserAuth..." functions to work same way, and now connections get rejected also when NO_FAILURE_ON_REJECT is not used.

This patch updates the other functions to follow same logic as DoUserAuthRequestNone() already does. Not sure if this is necessary "correct" solution, but seems to solve my problem...

[wolfSSL-Bot]

Can one of the admins verify this patch?

[tjko]

@dgarske, seems like there is/was some (macos-latest) workflow issue? As there seems to have been same failures on other PRs (like this one: https://github.com/wolfSSL/wolfssh/actions/runs/17214562015).

Btw, I don't seem to have permission to "Re-run jobs", so what would be best way to re-trigger workflows (once macos flows work again)?

[dgarske]

@dgarske, seems like there is/was some (macos-latest) workflow issue? As there seems to have been same failures on other PRs (like this one: https://github.com/wolfSSL/wolfssh/actions/runs/17214562015).

Btw, I don't seem to have permission to "Re-run jobs", so what would be best way to re-trigger workflows (once macos flows work again)?

@ejohnstown or @JacobBarthelmeh ?

[JacobBarthelmeh]

Hi @tjko,

Right, macos-latest updated recently to macos-15. GitHub Action's version of macos-15 was causing one of the tests some issues. This commit reverted the macos version for now (4f5f75f). To restart the test with the known good macos version rebase the work onto wolfssh/master.

git remote add wolfssh git@github.com:wolfssl/wolfssh
git fetch wolfssh
git rebase wolfssh//master
git remote rm wolfssh
git push -f

[tjko]

@JacobBarthelmeh, thanks. That worked great.

[JacobBarthelmeh]

@tjko I've been reviewing this. I think you are right that the behavior should be adjusted when encountering a WOLFSSH_USERAUTH_REJECTED return value. That return WOLFSSH_USERAUTH_REJECTED was added back around wolfSSH v1.4.2 for a password retry timeout option, and should stop the connection attempt when seen from the server.

I'd like to keep the SendUserAuthFailure calls limited to once per authorization type though (once per function). I've opened this PR with that adjustment (#837). Thank you for the report, I'm closing this out in favor of the other PR in process.