From c52a5eae032f8e5383a24283c9152c5a88c0ac5f Mon Sep 17 00:00:00 2001 From: Giedrius Grabauskas <43740166+Grabauskas@users.noreply.github.com> Date: Mon, 19 Jan 2026 12:10:35 +0200 Subject: [PATCH 1/8] Add Notifications Architecture documentation --- docs/notifications-architecture.md | 131 +++++++++++++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 docs/notifications-architecture.md diff --git a/docs/notifications-architecture.md b/docs/notifications-architecture.md new file mode 100644 index 0000000..08da7b3 --- /dev/null +++ b/docs/notifications-architecture.md @@ -0,0 +1,131 @@ +--- +title: "Notifications Architecture" +description: "Overview of the Notifications system architecture, detailing retry strategies with Polly and Hangfire, circuit breaker patterns, and HTTP status code handling." +keywords: [notifications, architecture, polly, hangfire, circuit breaker, retry, webhook, OneGround, ZGW] +--- + +# Notifications + +When creating, modifying, or deleting each ZGW entity, a notification is sent. Clients (client apps) can subscribe to these notifications. This requires a client (client app) webhook receiver to which the notifications can be delivered. The URL (and authentication) of this webhook receiver is stored in the client's subscription in our Notifications database. +There are four key components to the Notifications system: + +1. Polly retries +2. Hangfire retries and priority queue +3. Circuit breaker +4. Http codes that lead to retries + +## Polly retries + +If a notification cannot be delivered to a client webhook receiver, for example, because the service is (temporarily) down, a new attempt will be made to deliver the notification within a few seconds. Both the interval pattern (linear/exponential) and the number of attempts can be configured. An important aspect of Polly retry is that it is a blocking call, so it is important to ensure that the total retry sequence is not too long (<8 seconds). Another important aspect of Polly retry is that retries are not persistent (i.e., they take place in memory). A typical Polly retry looks like this: + +```json +{ + "PollyConfig": { + "NotificatiesSender": { + "Retry": { + "MaxRetryAttempts": 4, + "BackoffType": "Exponential", + "Delay": "00:00:01" + }, + "Timeout": { + "Timeout": "00:00:30" + } + } + } +} +``` + +## Hangfire retries and priority queue + +The second level of retries is based on the Hangfire Scheduler. It's possible that the client webhook receiver is down for an extended period. In this case, the Polly retry won't work. The Hangfire retry offers a solution because retries are scheduled and picked up and processed at a (much) later time, for example, after four hours or even after several days. Unlike Polly retries, Hangfire retries are persistent (stored as jobs in the Notification database). Hangfire retries use two queues: the MAIN and RETRY queues. New notifications are placed in the MAIN queue, while scheduled retries are placed in the RETRY queue. This prevents new notifications from having to wait until all retries have been processed, as only a limited number of jobs can be executed. More importantly, the retry period can be long, for example, one day. + +After the last failed retry, Hangfire will move the retry(job) to Failed Jobs. + +A nice feature is that Hangfire includes a Dashboard that displays all jobs (Retry Jobs, Successfully Executed Jobs, Failed Jobs, and Deleted Jobs). Failed jobs can even be restarted manually. + +A typical Hangfire retry configuration looks like this: + +```json +{ + "Hangfire": { + "RetrySchedule": "0.00:15;0.00:30;0.01:00;0.04:00;1.00:00", + "ExpireFailedJobsScanAt": "05:00", + "ExpireFailedJobAfter": "7.00:00" + } +} +``` + +The retry pattern (configured in RetrySchedule) is: + +- 15 minutes +- 30 minutes +- 1 hour +- 4 hours +- 1 day + +There are two other settings intended for automatically cleaning up failed retry jobs: + +- ExpireFailedJobsScanAt (scan interval or the value: 'never', 'disabled', 'n/a') +- ExpireFailedJobAfter (period that failed jobs must continue to exist) + +## Circuit breaker + +Another component of the notification system is the circuit breaker. The circuit breaker prevents unresponsive (or faulty) webhook receivers from repeatedly reporting an error over a period of time. This can severely and unnecessarily block the system (think of unnecessary timeouts). The idea is that a webhook receiver is only allowed to fail a limited number of times. If a webhook receiver fails, it is MONITORED, and the number of failures is recorded. For example, after 10 failures, the webhook receiver is marked as BLOCKED (and therefore no longer MONITORED). The circuit breaker maintains this block for a specified period, for example, 5 minutes. After this, it tries to deliver the notification again. If the receiver fails again (maximum 10 times), the block occurs again. This mechanism prevents unnecessary calls to unresponsive webhook receivers, which helps improve system performance. + +All webhook receiver blocks will be automatically released after a certain period of time (unless called within this time). + +A typical Circuit breaker configuration looks like this: + +```json +{ + "CircuitBreaker": { + "FailureThreshold": 10, + "BreakDuration": "00:05:00", + "CacheExpirationMinutes": 10 + } +} +``` + +The possible settings under CircuitBreaker are: + +- FailureThreshold (number of times you can fail before the BLOCKADE takes effect) +- BreakDuration (time of the BLOCKADE) +- CacheExpirationMinutes (time that MONITORING and BLOCKS are lifted unless called) + +## Http codes that lead to retries + +### Polly retry + +Polly will perform retries according to the configured policy for the following HTTP status codes: + +- HttpRequestException +- All HTTP 5xx codes +- 408 Request Timeout +- 429 TooManyRequests + +With the other HTTP status codes, the error is returned immediately. + +It's possible to perform a retry on HTTP status codes that aren't supported by Polly. This can be done using the AddRetryOnHttpStatusCodes setting directly under PollyConfig. Multiple codes are supported. So, if you want to perform a retry on a NotFound (404) error, you can configure this with the following line: + +```json +{ + "PollyConfig": { + "NotificatiesSender": { + "Retry": { + "MaxRetryAttempts": 4, + "BackoffType": "Exponential", + "UseJitter": true, + "Delay": "00:00:00.250" + }, + "Timeout": { + "Timeout": "00:00:30" + }, + "AddRetryOnHttpStatusCodes": [404] + } + } +} +``` + +### Hangfire retry + +Hangfire retry makes no distinction and will always perform a retry. From ad7eea66d729a7b43a5e2637c235f9ee9a51ea35 Mon Sep 17 00:00:00 2001 From: Giedrius Grabauskas <43740166+Grabauskas@users.noreply.github.com> Date: Mon, 19 Jan 2026 12:12:45 +0200 Subject: [PATCH 2/8] Refine Notifications Architecture documentation for clarity and consistency --- docs/notifications-architecture.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/notifications-architecture.md b/docs/notifications-architecture.md index 08da7b3..893e8ab 100644 --- a/docs/notifications-architecture.md +++ b/docs/notifications-architecture.md @@ -6,17 +6,18 @@ keywords: [notifications, architecture, polly, hangfire, circuit breaker, retry, # Notifications -When creating, modifying, or deleting each ZGW entity, a notification is sent. Clients (client apps) can subscribe to these notifications. This requires a client (client app) webhook receiver to which the notifications can be delivered. The URL (and authentication) of this webhook receiver is stored in the client's subscription in our Notifications database. +When a ZGW entity is created, modified, or deleted, a notification is sent. Client applications can subscribe to these notifications. This requires a webhook receiver on the client side to which the notifications can be delivered. The URL (and authentication) of this webhook receiver is stored in the client's subscription in our Notifications database. + There are four key components to the Notifications system: 1. Polly retries 2. Hangfire retries and priority queue 3. Circuit breaker -4. Http codes that lead to retries +4. HTTP status codes that lead to retries ## Polly retries -If a notification cannot be delivered to a client webhook receiver, for example, because the service is (temporarily) down, a new attempt will be made to deliver the notification within a few seconds. Both the interval pattern (linear/exponential) and the number of attempts can be configured. An important aspect of Polly retry is that it is a blocking call, so it is important to ensure that the total retry sequence is not too long (<8 seconds). Another important aspect of Polly retry is that retries are not persistent (i.e., they take place in memory). A typical Polly retry looks like this: +If a notification cannot be delivered to a client webhook receiver (for example, because the service is temporarily down), a new attempt is made to deliver the notification within a few seconds. Both the interval pattern (linear/exponential) and the number of attempts can be configured. An important aspect of Polly retries is that they are blocking calls; therefore, it is crucial to ensure that the total retry sequence is not too long (e.g., <8 seconds). Another key characteristic is that Polly retries are not persistent (i.e., they take place in memory). A typical Polly retry configuration looks like this: ```json { @@ -37,9 +38,9 @@ If a notification cannot be delivered to a client webhook receiver, for example, ## Hangfire retries and priority queue -The second level of retries is based on the Hangfire Scheduler. It's possible that the client webhook receiver is down for an extended period. In this case, the Polly retry won't work. The Hangfire retry offers a solution because retries are scheduled and picked up and processed at a (much) later time, for example, after four hours or even after several days. Unlike Polly retries, Hangfire retries are persistent (stored as jobs in the Notification database). Hangfire retries use two queues: the MAIN and RETRY queues. New notifications are placed in the MAIN queue, while scheduled retries are placed in the RETRY queue. This prevents new notifications from having to wait until all retries have been processed, as only a limited number of jobs can be executed. More importantly, the retry period can be long, for example, one day. +The second level of retries is based on the Hangfire Scheduler. It is possible that the client webhook receiver is down for an extended period. In this case, the Polly retry will not work. Hangfire retries offer a solution by scheduling retries to be processed at a later time—for example, after four hours or even several days. Unlike Polly retries, Hangfire retries are persistent (stored as jobs in the Notifications database). Hangfire retries use two queues: the MAIN and RETRY queues. New notifications are placed in the MAIN queue, while scheduled retries are placed in the RETRY queue. This prevents new notifications from waiting until all retries have been processed, as only a limited number of jobs can be executed continuously. More importantly, the retry period can be extended (e.g., up to one day). -After the last failed retry, Hangfire will move the retry(job) to Failed Jobs. +After the last failed retry, Hangfire moves the retry job to the 'Failed Jobs' state. A nice feature is that Hangfire includes a Dashboard that displays all jobs (Retry Jobs, Successfully Executed Jobs, Failed Jobs, and Deleted Jobs). Failed jobs can even be restarted manually. @@ -70,9 +71,11 @@ There are two other settings intended for automatically cleaning up failed retry ## Circuit breaker -Another component of the notification system is the circuit breaker. The circuit breaker prevents unresponsive (or faulty) webhook receivers from repeatedly reporting an error over a period of time. This can severely and unnecessarily block the system (think of unnecessary timeouts). The idea is that a webhook receiver is only allowed to fail a limited number of times. If a webhook receiver fails, it is MONITORED, and the number of failures is recorded. For example, after 10 failures, the webhook receiver is marked as BLOCKED (and therefore no longer MONITORED). The circuit breaker maintains this block for a specified period, for example, 5 minutes. After this, it tries to deliver the notification again. If the receiver fails again (maximum 10 times), the block occurs again. This mechanism prevents unnecessary calls to unresponsive webhook receivers, which helps improve system performance. +Another component of the notification system is the circuit breaker. This prevents the system from repeatedly attempting to contact unresponsive (or faulty) webhook receivers over a period of time, which can severely and unnecessarily block resources (due to timeouts). + +The concept is that a webhook receiver is only allowed to fail a limited number of times. When calls to a webhook receiver fail, the failures are monitored and recorded. For example, after 10 failures, the webhook receiver is marked as BLOCKED (and effectively no longer monitored). The circuit breaker maintains this block for a specified period (e.g., 5 minutes). After this period, it attempts to deliver the notification again. If the receiver fails again, the block is reapplied. This mechanism prevents unnecessary calls to unresponsive webhook receivers, improving system performance. -All webhook receiver blocks will be automatically released after a certain period of time (unless called within this time). +All webhook receiver blocks are automatically released after a specified period unless triggered again. A typical Circuit breaker configuration looks like this: @@ -92,16 +95,16 @@ The possible settings under CircuitBreaker are: - BreakDuration (time of the BLOCKADE) - CacheExpirationMinutes (time that MONITORING and BLOCKS are lifted unless called) -## Http codes that lead to retries +## HTTP status codes triggering retries ### Polly retry Polly will perform retries according to the configured policy for the following HTTP status codes: -- HttpRequestException +- `HttpRequestException` - All HTTP 5xx codes - 408 Request Timeout -- 429 TooManyRequests +- 429 Too Many Requests With the other HTTP status codes, the error is returned immediately. From d8cd307d3857c932492f21b3c515369df06c2c38 Mon Sep 17 00:00:00 2001 From: Giedrius Grabauskas <43740166+Grabauskas@users.noreply.github.com> Date: Mon, 19 Jan 2026 12:16:09 +0200 Subject: [PATCH 3/8] Fix HTML entity in Polly retries section of Notifications Architecture documentation --- docs/notifications-architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/notifications-architecture.md b/docs/notifications-architecture.md index 893e8ab..1c97aaf 100644 --- a/docs/notifications-architecture.md +++ b/docs/notifications-architecture.md @@ -17,7 +17,7 @@ There are four key components to the Notifications system: ## Polly retries -If a notification cannot be delivered to a client webhook receiver (for example, because the service is temporarily down), a new attempt is made to deliver the notification within a few seconds. Both the interval pattern (linear/exponential) and the number of attempts can be configured. An important aspect of Polly retries is that they are blocking calls; therefore, it is crucial to ensure that the total retry sequence is not too long (e.g., <8 seconds). Another key characteristic is that Polly retries are not persistent (i.e., they take place in memory). A typical Polly retry configuration looks like this: +If a notification cannot be delivered to a client webhook receiver (for example, because the service is temporarily down), a new attempt is made to deliver the notification within a few seconds. Both the interval pattern (linear/exponential) and the number of attempts can be configured. An important aspect of Polly retries is that they are blocking calls; therefore, it is crucial to ensure that the total retry sequence is not too long (e.g., <8 seconds). Another key characteristic is that Polly retries are not persistent (i.e., they take place in memory). A typical Polly retry configuration looks like this: ```json { From 68a4870c7d6b0e21c56fbb39c30941e13869ee94 Mon Sep 17 00:00:00 2001 From: heuvea <66989902+heuvea@users.noreply.github.com> Date: Mon, 19 Jan 2026 11:25:32 +0100 Subject: [PATCH 4/8] Update notifications-architecture.md --- docs/notifications-architecture.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/notifications-architecture.md b/docs/notifications-architecture.md index 1c97aaf..d6c379f 100644 --- a/docs/notifications-architecture.md +++ b/docs/notifications-architecture.md @@ -24,9 +24,10 @@ If a notification cannot be delivered to a client webhook receiver (for example, "PollyConfig": { "NotificatiesSender": { "Retry": { + "ShouldRetryAfterHeader": true, "MaxRetryAttempts": 4, "BackoffType": "Exponential", - "Delay": "00:00:01" + "Delay": "00:00:00.500" }, "Timeout": { "Timeout": "00:00:30" @@ -115,6 +116,7 @@ It's possible to perform a retry on HTTP status codes that aren't supported by P "PollyConfig": { "NotificatiesSender": { "Retry": { + "ShouldRetryAfterHeader": true, "MaxRetryAttempts": 4, "BackoffType": "Exponential", "UseJitter": true, @@ -123,7 +125,7 @@ It's possible to perform a retry on HTTP status codes that aren't supported by P "Timeout": { "Timeout": "00:00:30" }, - "AddRetryOnHttpStatusCodes": [404] + "AddRetryOnHttpStatusCodes": "404;..." } } } From fab8ecf181c7642cdeaebcc735fed73f300a7248 Mon Sep 17 00:00:00 2001 From: heuvea <66989902+heuvea@users.noreply.github.com> Date: Mon, 19 Jan 2026 11:35:26 +0100 Subject: [PATCH 5/8] Modify retry delay to 500ms in notifications architecture Updated the delay configuration for retry settings in the notifications architecture. --- docs/notifications-architecture.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/notifications-architecture.md b/docs/notifications-architecture.md index d6c379f..105c90b 100644 --- a/docs/notifications-architecture.md +++ b/docs/notifications-architecture.md @@ -27,6 +27,7 @@ If a notification cannot be delivered to a client webhook receiver (for example, "ShouldRetryAfterHeader": true, "MaxRetryAttempts": 4, "BackoffType": "Exponential", + "UseJitter": false, "Delay": "00:00:00.500" }, "Timeout": { @@ -120,7 +121,7 @@ It's possible to perform a retry on HTTP status codes that aren't supported by P "MaxRetryAttempts": 4, "BackoffType": "Exponential", "UseJitter": true, - "Delay": "00:00:00.250" + "Delay": "00:00:00.500" }, "Timeout": { "Timeout": "00:00:30" From 0cfd86322bdabfcd2e6b9283261782d9e1e22f96 Mon Sep 17 00:00:00 2001 From: heuvea <66989902+heuvea@users.noreply.github.com> Date: Mon, 19 Jan 2026 11:45:42 +0100 Subject: [PATCH 6/8] Enhance notifications architecture documentation Added details about Polly and Hangfire retry mechanisms for notifications. --- docs/notifications-architecture.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/notifications-architecture.md b/docs/notifications-architecture.md index 105c90b..8d6e828 100644 --- a/docs/notifications-architecture.md +++ b/docs/notifications-architecture.md @@ -38,6 +38,13 @@ If a notification cannot be delivered to a client webhook receiver (for example, } ``` +The Polly retries are performed at the following times: + +- 500 msec. +- 1 sec. +- 2 sec. +- 4 sec. + ## Hangfire retries and priority queue The second level of retries is based on the Hangfire Scheduler. It is possible that the client webhook receiver is down for an extended period. In this case, the Polly retry will not work. Hangfire retries offer a solution by scheduling retries to be processed at a later time—for example, after four hours or even several days. Unlike Polly retries, Hangfire retries are persistent (stored as jobs in the Notifications database). Hangfire retries use two queues: the MAIN and RETRY queues. New notifications are placed in the MAIN queue, while scheduled retries are placed in the RETRY queue. This prevents new notifications from waiting until all retries have been processed, as only a limited number of jobs can be executed continuously. More importantly, the retry period can be extended (e.g., up to one day). From 3dabc47028e3546655c6d75f3d714c26cd155b60 Mon Sep 17 00:00:00 2001 From: Johannes Battjes Date: Wed, 21 Jan 2026 12:05:21 +0100 Subject: [PATCH 7/8] johannes changes --- docs/gebruik-van-subscriptions-in-autorisaties.md | 12 ++++++------ docs/notifications-architecture.md | 12 ++++++------ sidebars.ts | 5 +++++ 3 files changed, 17 insertions(+), 12 deletions(-) diff --git a/docs/gebruik-van-subscriptions-in-autorisaties.md b/docs/gebruik-van-subscriptions-in-autorisaties.md index fc7451c..0563664 100644 --- a/docs/gebruik-van-subscriptions-in-autorisaties.md +++ b/docs/gebruik-van-subscriptions-in-autorisaties.md @@ -39,7 +39,7 @@ Filters allow your application to limit the notifications it receives to only th | Filter Key | Description | Allowed/Example Values | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `#resource` | The resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | zaak, status, zaakobject, zaakinformatieobject, zaakeigenschap, rol, resultaat, zaakbesluit | -| `#action` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | +| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | | `bronorganisatie` | The rsin of the organization that initiated or owns the case in the field bronorganisatie | 000001375 | | `zaaktype` | URL's of the casetype versions of the case in the field zaaktype | https://ztc.zgw.nl/api/v1/zaaktypen/b1fac1a1-7117-1e50-1d01-d155a715f1ed | | `vertrouwelijkheidaanduiding` | The confidentiality indication of the case in field vertrouwelijkheidaanduiding | openbaar, beperkt_openbaar, intern, zaakvertrouwelijk, vertrouwelijk, confidentieel, geheim, zeer_geheim | @@ -56,7 +56,7 @@ Filters allow your application to limit the notifications it receives to only th | Filter Key | Description | Allowed/Example Values | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | besluit, besluitinformatieobject | -| `#action` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | +| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | | `besluittype` | URL of the decision type version in the field besluittype | https://ztc.zgw.nl/api/v1/besluittypen/ce571dae-f0c1-a1e5-115a-f1acc1d171e5 | | `verantwoordelijke_organisatie` | Rsin of the organisation responsible for the decision in field verantwoordelijkeOrganisatie | 010110100 | | `besluittype_omschrijving` | The decision type in field omschrijving of besluittype | Beschikken op aanvraag | @@ -68,7 +68,7 @@ Filters allow your application to limit the notifications it receives to only th | Filter Key | Description | Allowed/Example Values | | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | enkelvoudiginformatieobject, gebruiksrechten, verzending | -| `#action` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | +| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | | `bronorganisatie` | The rsin of the organization that initiated or received and owns the document in the field bronorganisatie | 813264571 | | `informatieobjecttype` | URL of the information object type version in field informatieobjecttype | https://ztc.zgw.nl/api/v1/informatieobjecttypen/cadd1ce5-5a1a-7070-dabb-c1a551f1ab1e | | `vertrouwelijkheidaanduiding` | Confidentiality indication in field vertrouwelijkheidaanduiding | openbaar, beperkt_openbaar, intern, zaakvertrouwelijk, vertrouwelijk, confidentieel, geheim, zeer_geheim | @@ -83,7 +83,7 @@ Filters allow your application to limit the notifications it receives to only th | Filter Key | Description | Allowed/Example Values | | ----------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | zaaktype | -| `#action` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | +| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | | `catalogus` | The URL of the catalog in field catalogus | https://ztc.zgw.nl/api/v1/catalogussen/fe0ff0r5-fdd1-5011-1177-d15ac1d1f1ed | | `domein` | The domain of the catalog in field domein of catalogus | VTH | @@ -92,7 +92,7 @@ Filters allow your application to limit the notifications it receives to only th | Filter Key | Description | Allowed/Example Values | | ----------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | informatieobjecttype | -| `#action` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | +| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | | `catalogus` | The URL of the catalog in field catalogus | https://ztc.zgw.nl/api/v1/catalogussen/fe0ff0r5-fdd1-5011-1177-d15ac1d1f1ed | | `domein` | The domain of the catalog in field domein of catalogus | VTH | @@ -101,7 +101,7 @@ Filters allow your application to limit the notifications it receives to only th | Filter Key | Description | Allowed/Example Values | | ----------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | | `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | besluittype | -| `#action` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | +| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | | `catalogus` | The URL of the catalog in field catalogus | https://.../catalogi/fe0ff0r5-fdd1-5011-1177-d15ac1d1f1ed | | `domein` | The domain of the catalog in field domein of catalogus | VTH | diff --git a/docs/notifications-architecture.md b/docs/notifications-architecture.md index 8d6e828..ca78964 100644 --- a/docs/notifications-architecture.md +++ b/docs/notifications-architecture.md @@ -1,10 +1,10 @@ --- -title: "Notifications Architecture" +title: "How does the retry system for notitifations work?" description: "Overview of the Notifications system architecture, detailing retry strategies with Polly and Hangfire, circuit breaker patterns, and HTTP status code handling." keywords: [notifications, architecture, polly, hangfire, circuit breaker, retry, webhook, OneGround, ZGW] --- -# Notifications +# How does the retry system for notifications work? When a ZGW entity is created, modified, or deleted, a notification is sent. Client applications can subscribe to these notifications. This requires a webhook receiver on the client side to which the notifications can be delivered. The URL (and authentication) of this webhook receiver is stored in the client's subscription in our Notifications database. @@ -47,7 +47,7 @@ The Polly retries are performed at the following times: ## Hangfire retries and priority queue -The second level of retries is based on the Hangfire Scheduler. It is possible that the client webhook receiver is down for an extended period. In this case, the Polly retry will not work. Hangfire retries offer a solution by scheduling retries to be processed at a later time—for example, after four hours or even several days. Unlike Polly retries, Hangfire retries are persistent (stored as jobs in the Notifications database). Hangfire retries use two queues: the MAIN and RETRY queues. New notifications are placed in the MAIN queue, while scheduled retries are placed in the RETRY queue. This prevents new notifications from waiting until all retries have been processed, as only a limited number of jobs can be executed continuously. More importantly, the retry period can be extended (e.g., up to one day). +The second level of retries is based on the Hangfire Scheduler. It is possible that the client webhook receiver is down for an extended period. In this case, the Polly retry will not work. Hangfire retries offer a solution by scheduling retries to be processed at a later time, for example, after four hours or even several days. Unlike Polly retries, Hangfire retries are persistent (stored as jobs in the Notifications database). Hangfire retries use two queues: the MAIN and RETRY queues. New notifications are placed in the MAIN queue, while scheduled retries are placed in the RETRY queue. This prevents new notifications from waiting until all retries have been processed, as only a limited number of jobs can be executed continuously. More importantly, the retry period can be extended (e.g., up to one day). After the last failed retry, Hangfire moves the retry job to the 'Failed Jobs' state. @@ -115,9 +115,9 @@ Polly will perform retries according to the configured policy for the following - 408 Request Timeout - 429 Too Many Requests -With the other HTTP status codes, the error is returned immediately. +When other HTTP status codes are returned, Polly stops immediately (but Hangfire will retry). -It's possible to perform a retry on HTTP status codes that aren't supported by Polly. This can be done using the AddRetryOnHttpStatusCodes setting directly under PollyConfig. Multiple codes are supported. So, if you want to perform a retry on a NotFound (404) error, you can configure this with the following line: +It's possible to configure retries on additional HTTP status codes. This can be done using the AddRetryOnHttpStatusCodes setting directly under PollyConfig. If, for example, you want to retry always on unauthorized (401) errors, you can configure this with the following line: ```json { @@ -141,4 +141,4 @@ It's possible to perform a retry on HTTP status codes that aren't supported by P ### Hangfire retry -Hangfire retry makes no distinction and will always perform a retry. +The hangfire retry jobs make no distinction between different response codes and will always perform a retry. diff --git a/sidebars.ts b/sidebars.ts index 2a8a2a2..98154ec 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -11,6 +11,11 @@ const sidebars: SidebarsConfig = { id: "gebruik-van-subscriptions-in-autorisaties", label: "Use of Subscriptions for Notifications" }, + { + type: "doc", + id: "notifications-architecture", + label: "How does the retry system for notitifations work?" + }, "version-header", "example-document-upload/example-document-upload", "ztc1_3problemsandsolutions" From e473a84cb06078b0d5663cca3bbeac43343bf545 Mon Sep 17 00:00:00 2001 From: Giedrius Grabauskas <43740166+Grabauskas@users.noreply.github.com> Date: Wed, 21 Jan 2026 17:25:54 +0200 Subject: [PATCH 8/8] Fixed formatting --- docs/gebruik-van-subscriptions-in-autorisaties.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/gebruik-van-subscriptions-in-autorisaties.md b/docs/gebruik-van-subscriptions-in-autorisaties.md index 0563664..c30dc14 100644 --- a/docs/gebruik-van-subscriptions-in-autorisaties.md +++ b/docs/gebruik-van-subscriptions-in-autorisaties.md @@ -39,7 +39,7 @@ Filters allow your application to limit the notifications it receives to only th | Filter Key | Description | Allowed/Example Values | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `#resource` | The resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | zaak, status, zaakobject, zaakinformatieobject, zaakeigenschap, rol, resultaat, zaakbesluit | -| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | +| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | | `bronorganisatie` | The rsin of the organization that initiated or owns the case in the field bronorganisatie | 000001375 | | `zaaktype` | URL's of the casetype versions of the case in the field zaaktype | https://ztc.zgw.nl/api/v1/zaaktypen/b1fac1a1-7117-1e50-1d01-d155a715f1ed | | `vertrouwelijkheidaanduiding` | The confidentiality indication of the case in field vertrouwelijkheidaanduiding | openbaar, beperkt_openbaar, intern, zaakvertrouwelijk, vertrouwelijk, confidentieel, geheim, zeer_geheim | @@ -56,7 +56,7 @@ Filters allow your application to limit the notifications it receives to only th | Filter Key | Description | Allowed/Example Values | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | besluit, besluitinformatieobject | -| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | +| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | | `besluittype` | URL of the decision type version in the field besluittype | https://ztc.zgw.nl/api/v1/besluittypen/ce571dae-f0c1-a1e5-115a-f1acc1d171e5 | | `verantwoordelijke_organisatie` | Rsin of the organisation responsible for the decision in field verantwoordelijkeOrganisatie | 010110100 | | `besluittype_omschrijving` | The decision type in field omschrijving of besluittype | Beschikken op aanvraag | @@ -68,7 +68,7 @@ Filters allow your application to limit the notifications it receives to only th | Filter Key | Description | Allowed/Example Values | | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | enkelvoudiginformatieobject, gebruiksrechten, verzending | -| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | +| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | | `bronorganisatie` | The rsin of the organization that initiated or received and owns the document in the field bronorganisatie | 813264571 | | `informatieobjecttype` | URL of the information object type version in field informatieobjecttype | https://ztc.zgw.nl/api/v1/informatieobjecttypen/cadd1ce5-5a1a-7070-dabb-c1a551f1ab1e | | `vertrouwelijkheidaanduiding` | Confidentiality indication in field vertrouwelijkheidaanduiding | openbaar, beperkt_openbaar, intern, zaakvertrouwelijk, vertrouwelijk, confidentieel, geheim, zeer_geheim | @@ -83,7 +83,7 @@ Filters allow your application to limit the notifications it receives to only th | Filter Key | Description | Allowed/Example Values | | ----------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | zaaktype | -| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | +| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | | `catalogus` | The URL of the catalog in field catalogus | https://ztc.zgw.nl/api/v1/catalogussen/fe0ff0r5-fdd1-5011-1177-d15ac1d1f1ed | | `domein` | The domain of the catalog in field domein of catalogus | VTH | @@ -92,7 +92,7 @@ Filters allow your application to limit the notifications it receives to only th | Filter Key | Description | Allowed/Example Values | | ----------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | informatieobjecttype | -| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | +| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | | `catalogus` | The URL of the catalog in field catalogus | https://ztc.zgw.nl/api/v1/catalogussen/fe0ff0r5-fdd1-5011-1177-d15ac1d1f1ed | | `domein` | The domain of the catalog in field domein of catalogus | VTH | @@ -101,7 +101,7 @@ Filters allow your application to limit the notifications it receives to only th | Filter Key | Description | Allowed/Example Values | | ----------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | | `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | besluittype | -| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | +| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy | | `catalogus` | The URL of the catalog in field catalogus | https://.../catalogi/fe0ff0r5-fdd1-5011-1177-d15ac1d1f1ed | | `domein` | The domain of the catalog in field domein of catalogus | VTH |