[Rate limit issue] Server Console -> Discord

[JaxkDev]

As much as the server console -> discord feature is wanted (note discord -> server console is in-dependant of this issue)
there are a few issues that need addressing for an efficient implementation.

Biggest issue is size + speed of consoles log, currently discords rate limit globally is:
1. 50 calls per second on all requests to the API

However there is an even stricter rate limit which is the focus of this issue,
2. 5 messages per 5 seconds per channel (TBC, Possibly 10 per 5s)

Also i would like to point out a third limitation that also applies to most implementations.
3. A message cannot be >8mb or >2000 characters in content.

This would clearly get smashed every time the server starts/stops or a stupid plugin that spams more then 4 things a tick (1/20th of a second)

There is a few ways to get around this but I've opened this issue for people to comment on RE these possible implementations before i get round to doing it (after API's finished+tested but probably before 2.0 release)

Possible ways of implementing:

1. Bulk Sending
   The main concept here is there is a 5s rate limit so all console logs will be stored in memory for 5s then joined by newline and sent as a single message meaning at most server log would be 1 message every 5s (safe from limit 2).
   However a limitation to this would be the single message could become >2000 characters (limit 3) a possible way around this is split the last 5s of stored messages into 2000character messages and send the first 4 and store any remaining to the front of the next 5s to be logged.

2. Log Files
   This ones fairly straight forward fairly recently discords pushed several visual updates one being the ability to view file content in discord without having to download or open the file locally therefor if the servers log was split into say 1min intervals or 5mb or 1hour (etc) we can send a file (<8mb) to discord with that time periods server log and that should be viewable in discord.

3. No Log
   Simply don't implement a way of sending servers log due to limitations. (don't plan to do this but left it here for peoples opinions)

4. Discarding Messages
   Once the ratelimit of 2000characters or x messages per second has been hit suffix the message with x messages discarded.
   Note this would require timeframes as the plugin cannot see into the future and know how many the server will log in the next second.

5. Embed Messages
   While embeds are still messages with limit 2 applying they allow for more content but at the cost of formatting.
   Limits on embeds:

• Embed titles are limited to 256 characters
• Embed descriptions are limited to 4096 characters
• There can be up to 25 fields
• A field's name is limited to 256 characters and its value to 1024 characters
• The footer text is limited to 2048 characters
• The author name is limited to 256 characters
• The sum of all characters from all embed structures in a message must not exceed 6000 characters

[Thunder33345]

All of this assumes that, logs MUST be as close to "realtime" as possible

Something here is being left out, you could simply do something like queued sending, for example cap the output speed at only 4.5 messages per second, the rest get queued for the next possible window

In this case, the ratelimit would never been hit, at cost of console not being realtime on high load periods like plugin being loaded

There's also the possibility of giving up a 1:1 console -> discord replica, by discarding messages if the ratelimit is filled replacing with (x5 messages discarded)

One could also use discord embeds which support more characters then normal 2k plain raw messages, though they might not look as nice as plain messages

If you ask me, i think it queuing and bulk sending could do the job, assuming console isnt at constant load, if you want to cover all bases, auto discard older queue in favor of catching up would be necessary

[JaxkDev]

Something here is being left out, you could simply do something like queued sending, for example cap the output speed at only 4.5 messages per second, the rest get queued for the next possible window

This was the idea behind 1, but if you have constant high output from console it would never return or decrease time delay and eventually could be hours behind, increasing memory and time lag hence the squash into 1 message to fit as much as possible it doesnt completely remove that risk but greatly reduces it.

There's also the possibility of giving up a 1:1 console -> discord replica, by discarding messages if the ratelimit is filled replacing with (x5 messages discarded)
Im not personally a fan of 'discarding' messages because quite simply it discards it lol, If your wanting to send console log to discord I'd have to assume the user is looking for a quick view at console instead of having to dig through a 100mb file after logging into server, locating it etc.

But im more then happy to append it onto the list its a valid option, similar with embed ill add forgot about that increased limit.

[Thunder33345]

This was the idea behind 1

That's summarized at the end of it:

If you ask me, i think it queuing and bulk sending could do the job, assuming console isnt at constant load, if you want to cover all bases, auto discard older queue in favor of catching up would be necessary

I dont see how bulking everything as one is a loss, or downside

By default, it would bulk everything to 2k OR next possible window for ratelimit, IF there's more, it would queue it up, IF the oldest unsent log is >2min, dump everything except the last 2k messages and sent it, ensuring the queue never run too far off from realtime

[JaxkDev]

This was the idea behind 1

That's summarized at the end of it:

If you ask me, i think it queuing and bulk sending could do the job, assuming console isnt at constant load, if you want to cover all bases, auto discard older queue in favor of catching up would be necessary

I dont see how bulking everything as one is a loss, or downside

By default, it would bulk everything to 2k OR next possible window for ratelimit, IF there's more, it would queue it up, IF the oldest unsent log is >2min, dump everything except the last 2k messages and sent it, ensuring the queue never run too far off from realtime

Hmm yeah I need some limit to the queue as you've mentioned 2min seems fair, if its more then that its likely a high constant load and I can just notify the user that the bot cannot send the log in realtime due to high volume of text.
More then likely itd be plugin spam or initial startup/shutdown which shouldnt take more then 2min at all seems good to me.

[Thunder33345]

i have to acknowledge that, we are going far beyond the mark here

platforms like pterodatchy just drop logs if ratelimit is passed

messages that are lost? well it's what it is

not to say it in a way that discourages doing thing "properly" rather then saying "it is what it is"

[JaxkDev]

If im going to do something especially from scratch like this id prefer to do it right and efficient the first time round otherwise it will take me a very long time to refactor and do correctly hence why 2.0 is taking centuries lol
I know nothing will be perfect but you get the idea i hope.

[Thunder33345]

just notify the user that the bot cannot send the log in realtime due to high volume of text.

hmmm editing messages is an option, so you could append (Log in sending queue: 100 lines, in 2 batches, lag time 1min)

or something else to let user know the state of current log queue isnt empty, and be edited so only the last message in the channel displays the status, if there's higher then normal messages

[JaxkDev]

That’s very specific stats which could only be true for 1second before requiring another 2 changes / edits which in turn collectively adds up to limit 1, preferably send once and that’s it that seems like it’d need several edits to remain true at all times

…

On Mon, 21 Jun 2021 at 21:05, Thunder33345 ***@***.***> wrote: just notify the user that the bot cannot send the log in realtime due to high volume of text. hmmm editing messages is an option, so you could append (Log in sending queue: 100 lines, in 2 batches, lag time 1min) or something else to let user know the state of current log queue isnt empty, and be edited so only the last message in the channel displays the status, if there's higher then normal messages — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/JaxkDev/DiscordBot/issues/28#issuecomment-865308930>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGFVMIDFOPI22VUII5R2NN3TT6LR5ANCNFSM47CGGHWQ> .

[Thunder33345]

yeah could just be left as it is wiithout update

so it would just have Logger Queue: X in every sent message, and only "updates" when it resends a new bulk message

[JaxkDev]

Pending move to related repo.