Open Function https://status.openfn.org/ 2026-04-17T04:09:17+00:00 Open Function <![CDATA[Shut down workers temporarily]]> https://status.openfunction.io/incident/37/ We've shut down workers while investigation a bug with Oauth and will be back online with more information shortly. Data sent to OpenFn will not be lost, but runs will not be run until we're back online.

Update: Workers are back online. Still investigating a bug with duplicate Oauth credentials per user; see Friday's product updates for details: https://community.openfn.org/c/product-updates/20

]]> 2025-06-19T12:30:00+00:00 Open Function <![CDATA[Runs temporarily paused]]> https://status.openfunction.io/incident/35/ No data is being lost, but runs are temporarily not being run on app.openfn.org . This incident started at 9:30pm on Jun 10th.

Update: June 10th - 07:20am UTC - we've had to isolate several heavy-load projects, but smaller projects are now back running.

Update: June 10th - 8:00am UTC - all projects are back online, some queues are still quite large if project owners have reduced their concurrency setting

]]> 2025-06-10T21:30:00+00:00 Open Function <![CDATA[500s on inbox endpoint, downtime to resolve urgent database issue]]> https://status.openfunction.io/incident/34/ We detected 500s on the inbox API endpoint earlier this morning, and had to take down app.openfn.org for 4 minutes and 35 seconds today to resolve a critical DB issue. We're back online and don't anticipate any further issues. By 11:01 AM UTC we were back online.

]]> 2025-02-04T10:56:00+00:00 Open Function <![CDATA[Email notifications not working]]> https://status.openfunction.io/incident/33/ Email notifications are not working due to an issue with our 3rd-party mailing service. We are currently working with them to resolve the issue.

Update: Email notifications are all working properly again.

Email notifications were not sent from 9am UTC on August 9th until 6:08pm on August 10th.

]]> 2024-08-09T09:00:00+00:00 Open Function <![CDATA[Community forum down]]> https://status.openfunction.io/incident/32/ Community forum down after upgrade. Fixing now.

Update at 802am UTC: Back online

]]> 2024-06-21T07:32:00+00:00 Open Function <![CDATA[Certificate-based unavailability]]> https://status.openfunction.io/incident/31/ Certificate warning displayed for ~40 minutes

]]> 2024-05-12T11:40:00+00:00 Open Function <![CDATA[DNS issues on openfn.org]]> https://status.openfunction.io/incident/30/ Experiencing DNS problems on openfn.org - will report back when in the clear

Update at 12:04:00 - all good.

]]> 2024-02-09T10:00:00+00:00 Open Function <![CDATA[Planned database maintenance]]> https://status.openfunction.io/incident/29/ UPDATE: Database maintenance finished.

We will be performing database maintenance tonight at 11pm UTC and expect about 60 seconds of disruption on inbox endpoints.

]]> 2023-04-28T23:00:00+00:00 Open Function <![CDATA[Data sent to naked domain (https://openfn.org/inbox...) not reaching inboxes]]> https://status.openfunction.io/incident/28/ ** Update at 15:10 UTC **

Fixed via updated DNS records

** Initial issue detected **

We're investigating a potential issue related to naked domain redirects. If you have a form fordwarding or webhook notification service set up to send data to an OpenFn inbox that does NOT contain "www" you may not be able to send data to OpenFn right now. Change your form forwarding / webhook noticiation address to include the "www" as below...

FROM: https://openfn.org/inbox/a8c9dd49-secret-uuid
TO:   https://www.openfn.org/inbox/a8c9dd49-secret-uuid
]]> 2023-04-17T14:18:00+00:00 Open Function <![CDATA[Slow processing time for new workorders]]> https://status.openfunction.io/incident/27/ Our team is monitoring an usually large spike in activity that's causing delays (expected, through a queuing system) between when a message is received to start a realtime workflow and when the first job begins. No payload have been dropped. Will update soon.

]]> 2023-04-01T11:28:00+00:00 Open Function <![CDATA[502s on US and Swiss servers]]> https://status.openfunction.io/incident/26/ From 2pm to 4pm UTC today we detected a number of 502s on our US and Swiss servers due to a deployment rollout error. This issue has been fixed.

]]> 2022-04-15T14:00:00+00:00 Open Function <![CDATA[Outage during new deployment]]> https://status.openfunction.io/incident/25/ We had a brief outage during the rollout of a new deployment. It lasted approximately 120 seconds. The cause was likely a node scaledown event alongside an incorrectly configured pod-distruption-budget which has now been addressed.

]]> 2022-03-29T11:31:00+00:00 Open Function <![CDATA[502s on Swiss Servers]]> https://status.openfunction.io/incident/24/ At 13:00 UTC we've marked this as fully resolved.

At 12:20 UTC we detected 502s on Swiss servers that may have been related to a Kubernetes version change. We have made a fix and service is restored, but we are still monitoring the load balancer and Kubernetes cluster.

]]> 2021-11-19T12:20:00+00:00 Open Function <![CDATA[GCP Outage]]> https://status.openfunction.io/incident/23/ The GCP outage is affecting our Swiss servers as well.

UPDATE: GCP is back!

]]> 2021-11-16T17:55:00+00:00 Open Function <![CDATA[GCP Outage]]> https://status.openfunction.io/incident/22/ There is a large-scale GPC outage affecting multiple Google Cloud Platform customers (incl. Spotify, Etsy, Zenhub, etc.). Our US Servers are hosted on GCP and currently unavailable due to the GCP outage.

UPDATE: GCP is back!

]]> 2021-11-16T17:42:00+00:00 Open Function <![CDATA[Incomplete logs for outputs with special characters]]> https://status.openfunction.io/incident/21/ While job running is not affecting, our new logging agent is cutting of final logs early if they contain certain special characters. These may be characters in other alphabets. We've reproduced the issue, confirmed that it doesn't affect the run itself (only the logs) and are currently working on a fix for the logging agent.

UPDATE 15:20:36 UTC - We've fixed the issue are are currently testing on our staging servers. If all goes well, logs with special characters will not be truncated very soon. Will update shortly.

RESOLUTION 15:50:00 UTC - The fix has been rolled out across all US and Swiss servers. Some run logs with special characters appeared truncated between 11:00 and 15:50 UTC today but no runs were affected.

]]> 2021-03-29T11:00:00+00:00 Open Function <![CDATA[502s on /inbox endpoint]]> https://status.openfunction.io/incident/20/ Between 10:20am and 7:23pm we saw 502s related to a load balancer issue and increased traffic on the /inbox/ endpoints on the US servers. This issue has been resolved.

]]> 2021-03-17T10:20:00+00:00 Open Function <![CDATA[Metrics grouping incorrectly on the status page. (This page!)]]> https://status.openfunction.io/incident/19/ Certain metrics have been grouping innacurately, making job runner availability appear to be clustered in 5 minute intervals when it's actually tested every 60 seconds. We're investigating the bug right now, but it's an issue with our status page (this page) rather than the job runners themselves.

Update: The grouping issue is fixed.

]]> 2020-07-09T01:05:00+00:00 Open Function <![CDATA[SSL Issues only for Swiss servers (message-triggered jobs)]]> https://status.openfunction.io/incident/18/ We're experience SSL certificate issues on our swiss servers. Working on the fix now. Timer and flow-triggered jobs are NOT affected.

UPDATE: The SSL certificate error has been fixed.

]]> 2020-06-18T17:44:00+00:00 Open Function <![CDATA[Planned Database Maintenance For Swiss Servers]]> https://status.openfunction.io/incident/17/ The swiss servers are undergoing planned database upgrades and all active Swiss-server users have been notified.

The scheduled downtime took 10 minutes exactly. We upgraded our Swiss servers to Postgres 12, enabling Better performance, security, and streaming replication so that future upgrades will not require downtime.

]]> 2020-06-16T01:00:00+00:00 Open Function <![CDATA[Inbox and job runner errors]]> https://status.openfunction.io/incident/16/ From 01:54 to 02:12 UTC and again from 02:19 to 03:56 UTC we experienced degraded service on the inbox and job runners. Messages posted to OpenFn during this time may not have been received and timed jobs scheduled during this time may not have run. Please check your inbox and run history pages to verify and trigger any retries/job runs as needed.

]]> 2020-04-26T01:54:00+00:00 Open Function <![CDATA[Status monitoring issues, a small number of 500s on inbox]]> https://status.openfunction.io/incident/15/ We had metric delivery issues with the status page and threw a small number of 500s on the inbox endpoint between 16:30 GMT and 17:05pm GMT

]]> 2020-01-29T16:34:00+00:00 Open Function <![CDATA[Delays in run status display]]> https://status.openfunction.io/incident/14/ This morning we identified and fixed a problem affecting a small number of users that lead to runs not being displayed as finished until several hours after they had completed. The issue is resolved.

]]> 2019-11-30T02:37:00+00:00 Open Function <![CDATA[Server returning 500s, now fixed]]> https://status.openfunction.io/incident/13/ For approximately 30 minutes www.openfn.org returned 500s. The issue has been resolved.

]]> 2019-11-16T15:45:00+00:00 Open Function <![CDATA[False interruption displayed on status.openfn.org]]> https://status.openfunction.io/incident/12/ The status of our Swiss servers was reported as interrupted by our metrics system, while it was actually in working order. This has been addressed and the metrics are now up-to-date.

]]> 2019-10-21T10:08:00+00:00 Open Function <![CDATA[Upgrading storage]]> https://status.openfunction.io/incident/11/ https://twitter.com/openfnstatus/status/1172500873660026884?s=19

]]> 2019-09-13T14:24:00+00:00 Open Function <![CDATA[Unscheduled downtime due to GCP billing transition]]> https://status.openfunction.io/incident/10/ From 1838 to 1938 GMT we had unplanned downtime due to a billing changeover at Google Cloud. Full service has been restored.

]]> 2019-05-23T19:38:00+00:00 Open Function <![CDATA[Scaling servers]]> https://status.openfunction.io/incident/9/ OpenFn had approximately 2 minutes of downtime while performing server scaling.

]]> 2019-05-05T18:35:00+00:00 Open Function <![CDATA[Status reporting issues, no platform issues at the moment.]]> https://status.openfunction.io/incident/8/ Update, 7:51am GMT - This status testing issue has been resolved. The message throughput testing mechanism has some minor performance issues at the moment—the core platform remains unaffected.

]]> 2019-03-09T07:46:00+00:00 Open Function <![CDATA[Slowdowns in our message processing]]> https://status.openfunction.io/incident/7/ 18:07 GMT - performance issues are over. Everything operating as usual.

We're currently experiences slowdowns in our message processing queue due to high server load. Please excuse the delay.

]]> 2019-03-06T15:10:00+00:00 Open Function <![CDATA[DB Connection errors]]> https://status.openfunction.io/incident/6/ Fix deployed
  • We're currently watching a database connection error that we've addressed with a fix.

Update Feb 20th, 16:27 GMT

  • Availability was not sacrificed, but individual job runs may have failed. Reach out to support@openfn.org if you have any questions.

Update Feb 20th, 18:09 GMT

  • All quiet since the fix, status metrics display proper performance. Marking fixed.
]]> 2019-02-20T14:14:00+00:00 Open Function <![CDATA[500s on inbound messages]]> https://status.openfunction.io/incident/5/ OpenFn threw 500s on inbound messages from 6am to 10am GMT. The bug has been identified and resolved.

]]> 2018-08-25T06:00:00+00:00 Open Function <![CDATA[Server Upgrades]]> https://status.openfunction.io/incident/4/ At 21:00:00 UTC we moved OpenFn to larger servers. We were offline for 3 minutes.

]]> 2018-04-07T21:00:00+00:00 Open Function <![CDATA[Slowness on the job running]]> https://status.openfunction.io/incident/3/ Update: the problem has been resolved and there are no more delays.

We're experiencing delays with job running this morning. Jobs are still being processes but there may be a several minute lag until the issue is resolved.

]]> 2018-03-26T06:56:00+00:00 Open Function <![CDATA[Enhancing Status Page]]> https://status.openfunction.io/incident/2/ We are currently enhancing the monitoring services and performance metrics used to track OpenFn's health. This page (status.openfunction.io) will not be fully available for several more hours.

]]> 2018-03-16T12:10:00+00:00 Open Function <![CDATA[Delays in inbound message processing]]> https://status.openfunction.io/incident/1/ From 14:27 to 18:57 (UTC) inbound messages were not processed. All messages were properly stored in an inbound queue and processed beginning at 18:57 UTC. Service is now back to 100% functional and the inbound message delays are gone.

]]> 2018-03-15T14:27:00+00:00