Cloudi-Fi - Captive Portal services in the APAC region are impacted – Incident details

Background:

1. A maintenance operation was performed on Monday in the APAC environment.

https://status.cloudi-fi.net/cm9cj09s00001exvbas2uckxn

2. This operation included changes to the environment variables, notably for our service in charge of caching.

Incident Description:

1. Our server in APAC was incorrectly configured due to an improper set of environment variables.

2. Our caching service parameter used was the one from the European environment, while APAC now has its own caching endpoint.

3. This incorrect parameter only took effect after the new release was deployed on Tuesday (15 April 2025).

4. As a result, the APAC environment experienced service degradation or unavailability due to a failed Caching connection.

Root Cause:

1. The reference file containing environment variables for the APAC region was outdated and incorrect.

2. It led to the reuse of the European parameter, which is no longer valid for the APAC setup following the separation of environments.

Immediate Actions Taken:

1. Identified the misconfiguration through logs and connection error analysis.

2. Manually corrected the Redis parameter for the APAC environment.

3. Restarted impacted services to apply the corrected configuration.

Action Plan:

1. Complete review of the environment variables file for APAC.

2. Create and validate separate environment variable files for each region (Europe/APAC).

3. Implement a validation checklist before any multi-region deployment.

4. Audit recent deployments to ensure no other misconfigurations are present.

The configuration has been corrected and the services have returned to their normal stage. We are still monitoring the services.

The issue has been identified and is related to an incorrect routing on one of our nodes in APAC. This happened due to the incorrect configuration applied after the server was restarted.