Skip to content

Add Notifications Architecture documentation #89

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Grabauskas wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Add Notifications Architecture documentation #89

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c52a5ea
Add Notifications Architecture documentation
Grabauskas Jan 19, 2026
ad7eea6
Refine Notifications Architecture documentation for clarity and consi…
Grabauskas Jan 19, 2026
d8cd307
Fix HTML entity in Polly retries section of Notifications Architectur…
Grabauskas Jan 19, 2026
68a4870
Update notifications-architecture.md
heuvea Jan 19, 2026
fab8ecf
Modify retry delay to 500ms in notifications architecture
heuvea Jan 19, 2026
0cfd863
Enhance notifications architecture documentation
heuvea Jan 19, 2026
16902fb
Merge branch 'main' into docs/notifications
Grabauskas Jan 20, 2026
3dabc47
johannes changes
johannesbattjes Jan 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/gebruik-van-subscriptions-in-autorisaties.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Filters allow your application to limit the notifications it receives to only th
| Filter Key | Description | Allowed/Example Values |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `#resource` | The resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | zaak, status, zaakobject, zaakinformatieobject, zaakeigenschap, rol, resultaat, zaakbesluit |
| `#action` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy |
| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy |
| `bronorganisatie` | The rsin of the organization that initiated or owns the case in the field bronorganisatie | 000001375 |
| `zaaktype` | URL's of the casetype versions of the case in the field zaaktype | https://ztc.zgw.nl/api/v1/zaaktypen/b1fac1a1-7117-1e50-1d01-d155a715f1ed |
| `vertrouwelijkheidaanduiding` | The confidentiality indication of the case in field vertrouwelijkheidaanduiding | openbaar, beperkt_openbaar, intern, zaakvertrouwelijk, vertrouwelijk, confidentieel, geheim, zeer_geheim |
Expand All @@ -56,7 +56,7 @@ Filters allow your application to limit the notifications it receives to only th
| Filter Key | Description | Allowed/Example Values |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | besluit, besluitinformatieobject |
| `#action` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy |
| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy |
| `besluittype` | URL of the decision type version in the field besluittype | https://ztc.zgw.nl/api/v1/besluittypen/ce571dae-f0c1-a1e5-115a-f1acc1d171e5 |
| `verantwoordelijke_organisatie` | Rsin of the organisation responsible for the decision in field verantwoordelijkeOrganisatie | 010110100 |
| `besluittype_omschrijving` | The decision type in field omschrijving of besluittype | Beschikken op aanvraag |
Expand All @@ -68,7 +68,7 @@ Filters allow your application to limit the notifications it receives to only th
| Filter Key | Description | Allowed/Example Values |
| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | enkelvoudiginformatieobject, gebruiksrechten, verzending |
| `#action` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy |
| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy |
| `bronorganisatie` | The rsin of the organization that initiated or received and owns the document in the field bronorganisatie | 813264571 |
| `informatieobjecttype` | URL of the information object type version in field informatieobjecttype | https://ztc.zgw.nl/api/v1/informatieobjecttypen/cadd1ce5-5a1a-7070-dabb-c1a551f1ab1e |
| `vertrouwelijkheidaanduiding` | Confidentiality indication in field vertrouwelijkheidaanduiding | openbaar, beperkt_openbaar, intern, zaakvertrouwelijk, vertrouwelijk, confidentieel, geheim, zeer_geheim |
Expand All @@ -83,7 +83,7 @@ Filters allow your application to limit the notifications it receives to only th
| Filter Key | Description | Allowed/Example Values |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | zaaktype |
| `#action` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy |
| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy |
| `catalogus` | The URL of the catalog in field catalogus | https://ztc.zgw.nl/api/v1/catalogussen/fe0ff0r5-fdd1-5011-1177-d15ac1d1f1ed |
| `domein` | The domain of the catalog in field domein of catalogus | VTH |

Expand All @@ -92,7 +92,7 @@ Filters allow your application to limit the notifications it receives to only th
| Filter Key | Description | Allowed/Example Values |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | informatieobjecttype |
| `#action` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy |
| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy |
| `catalogus` | The URL of the catalog in field catalogus | https://ztc.zgw.nl/api/v1/catalogussen/fe0ff0r5-fdd1-5011-1177-d15ac1d1f1ed |
| `domein` | The domain of the catalog in field domein of catalogus | VTH |

Expand All @@ -101,7 +101,7 @@ Filters allow your application to limit the notifications it receives to only th
| Filter Key | Description | Allowed/Example Values |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| `#resource` | the resources as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | besluittype |
| `#action` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy |
| `#actie` | The actions as listed in the ZGW Open Api Specification, and presented in the resource field in the notification message itself | create, update, destroy |
| `catalogus` | The URL of the catalog in field catalogus | https://.../catalogi/fe0ff0r5-fdd1-5011-1177-d15ac1d1f1ed |
| `domein` | The domain of the catalog in field domein of catalogus | VTH |

Expand Down
144 changes: 144 additions & 0 deletions docs/notifications-architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
---
title: "How does the retry system for notitifations work?"
description: "Overview of the Notifications system architecture, detailing retry strategies with Polly and Hangfire, circuit breaker patterns, and HTTP status code handling."
keywords: [notifications, architecture, polly, hangfire, circuit breaker, retry, webhook, OneGround, ZGW]
---

# How does the retry system for notifications work?

When a ZGW entity is created, modified, or deleted, a notification is sent. Client applications can subscribe to these notifications. This requires a webhook receiver on the client side to which the notifications can be delivered. The URL (and authentication) of this webhook receiver is stored in the client's subscription in our Notifications database.

There are four key components to the Notifications system:

1. Polly retries
2. Hangfire retries and priority queue
3. Circuit breaker
4. HTTP status codes that lead to retries

## Polly retries

If a notification cannot be delivered to a client webhook receiver (for example, because the service is temporarily down), a new attempt is made to deliver the notification within a few seconds. Both the interval pattern (linear/exponential) and the number of attempts can be configured. An important aspect of Polly retries is that they are blocking calls; therefore, it is crucial to ensure that the total retry sequence is not too long (e.g., <8 seconds). Another key characteristic is that Polly retries are not persistent (i.e., they take place in memory). A typical Polly retry configuration looks like this:

```json
{
"PollyConfig": {
"NotificatiesSender": {
"Retry": {
"ShouldRetryAfterHeader": true,
"MaxRetryAttempts": 4,
"BackoffType": "Exponential",
"UseJitter": false,
"Delay": "00:00:00.500"
},
"Timeout": {
"Timeout": "00:00:30"
}
}
}
}
```

The Polly retries are performed at the following times:

- 500 msec.
- 1 sec.
- 2 sec.
- 4 sec.

## Hangfire retries and priority queue

The second level of retries is based on the Hangfire Scheduler. It is possible that the client webhook receiver is down for an extended period. In this case, the Polly retry will not work. Hangfire retries offer a solution by scheduling retries to be processed at a later time, for example, after four hours or even several days. Unlike Polly retries, Hangfire retries are persistent (stored as jobs in the Notifications database). Hangfire retries use two queues: the MAIN and RETRY queues. New notifications are placed in the MAIN queue, while scheduled retries are placed in the RETRY queue. This prevents new notifications from waiting until all retries have been processed, as only a limited number of jobs can be executed continuously. More importantly, the retry period can be extended (e.g., up to one day).

After the last failed retry, Hangfire moves the retry job to the 'Failed Jobs' state.

A nice feature is that Hangfire includes a Dashboard that displays all jobs (Retry Jobs, Successfully Executed Jobs, Failed Jobs, and Deleted Jobs). Failed jobs can even be restarted manually.

A typical Hangfire retry configuration looks like this:

```json
{
"Hangfire": {
"RetrySchedule": "0.00:15;0.00:30;0.01:00;0.04:00;1.00:00",
"ExpireFailedJobsScanAt": "05:00",
"ExpireFailedJobAfter": "7.00:00"
}
}
```

The retry pattern (configured in RetrySchedule) is:

- 15 minutes
- 30 minutes
- 1 hour
- 4 hours
- 1 day

There are two other settings intended for automatically cleaning up failed retry jobs:

- ExpireFailedJobsScanAt (scan interval or the value: 'never', 'disabled', 'n/a')
- ExpireFailedJobAfter (period that failed jobs must continue to exist)

## Circuit breaker

Another component of the notification system is the circuit breaker. This prevents the system from repeatedly attempting to contact unresponsive (or faulty) webhook receivers over a period of time, which can severely and unnecessarily block resources (due to timeouts).

The concept is that a webhook receiver is only allowed to fail a limited number of times. When calls to a webhook receiver fail, the failures are monitored and recorded. For example, after 10 failures, the webhook receiver is marked as BLOCKED (and effectively no longer monitored). The circuit breaker maintains this block for a specified period (e.g., 5 minutes). After this period, it attempts to deliver the notification again. If the receiver fails again, the block is reapplied. This mechanism prevents unnecessary calls to unresponsive webhook receivers, improving system performance.

All webhook receiver blocks are automatically released after a specified period unless triggered again.

A typical Circuit breaker configuration looks like this:

```json
{
"CircuitBreaker": {
"FailureThreshold": 10,
"BreakDuration": "00:05:00",
"CacheExpirationMinutes": 10
}
}
```

The possible settings under CircuitBreaker are:

- FailureThreshold (number of times you can fail before the BLOCKADE takes effect)
- BreakDuration (time of the BLOCKADE)
- CacheExpirationMinutes (time that MONITORING and BLOCKS are lifted unless called)

## HTTP status codes triggering retries

### Polly retry

Polly will perform retries according to the configured policy for the following HTTP status codes:

- `HttpRequestException`
- All HTTP 5xx codes
- 408 Request Timeout
- 429 Too Many Requests

When other HTTP status codes are returned, Polly stops immediately (but Hangfire will retry).

It's possible to configure retries on additional HTTP status codes. This can be done using the AddRetryOnHttpStatusCodes setting directly under PollyConfig. If, for example, you want to retry always on unauthorized (401) errors, you can configure this with the following line:

```json
{
"PollyConfig": {
"NotificatiesSender": {
"Retry": {
"ShouldRetryAfterHeader": true,
"MaxRetryAttempts": 4,
"BackoffType": "Exponential",
"UseJitter": true,
"Delay": "00:00:00.500"
},
"Timeout": {
"Timeout": "00:00:30"
},
"AddRetryOnHttpStatusCodes": "404;..."
}
}
}
```

### Hangfire retry

The hangfire retry jobs make no distinction between different response codes and will always perform a retry.
5 changes: 5 additions & 0 deletions sidebars.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@ const sidebars: SidebarsConfig = {
id: "gebruik-van-subscriptions-in-autorisaties",
label: "Use of Subscriptions for Notifications"
},
{
type: "doc",
id: "notifications-architecture",
label: "How does the retry system for notitifations work?"
},
"version-header",
"example-document-upload/example-document-upload",
"ztc1_3problemsandsolutions"
Expand Down
Loading