Cloudi-Fi - Notice history

Background:

1. A maintenance operation was performed on Monday in the APAC environment.

https://status.cloudi-fi.net/cm9cj09s00001exvbas2uckxn

2. This operation included changes to the environment variables, notably for our service in charge of caching.

Incident Description:

1. Our server in APAC was incorrectly configured due to an improper set of environment variables.

2. Our caching service parameter used was the one from the European environment, while APAC now has its own caching endpoint.

3. This incorrect parameter only took effect after the new release was deployed on Tuesday (15 April 2025).

4. As a result, the APAC environment experienced service degradation or unavailability due to a failed Caching connection.

Root Cause:

1. The reference file containing environment variables for the APAC region was outdated and incorrect.

2. It led to the reuse of the European parameter, which is no longer valid for the APAC setup following the separation of environments.

Immediate Actions Taken:

1. Identified the misconfiguration through logs and connection error analysis.

2. Manually corrected the Redis parameter for the APAC environment.

3. Restarted impacted services to apply the corrected configuration.

Action Plan:

1. Complete review of the environment variables file for APAC.

2. Create and validate separate environment variable files for each region (Europe/APAC).

3. Implement a validation checklist before any multi-region deployment.

4. Audit recent deployments to ensure no other misconfigurations are present.

The configuration has been corrected and the services have returned to their normal stage. We are still monitoring the services.

The issue has been identified and is related to an incorrect routing on one of our nodes in APAC. This happened due to the incorrect configuration applied after the server was restarted.

The root cause was the mis-handling of devices at the dhcp level leading to them not being associated with a pool. Only guest devices were affected. This mis-handling was introduced with the latest release (see https://status.cloudi-fi.net/cm9cj09s00001exvbas2uckxn)

Since the workaround has been applied, everything appears to be functioning as expected.

Devices are now able to obtain IP addresses from the appropriate DHCP pools without issue.

However, please note that a permanent fix is still to be implemented. Our engineering team continues to work on a long-term solution to fully address the root cause.