Workers on Packit Service Status

[Resolved] Maintenance outage

Mon, 03 Feb 2025 15:15:00 +0100

Today’s weekly production deployment requires a smaller outage to ensure proper redeployment without any inconsistencies.

We expect an outage for up to 15 minutes. There will be no jobs processed from 2025-02-03 15:15 UTC till the successful redeployment.

In case you are affected, please retrigger the jobs accordingly (via comments, or Re-Run button on GitHub Checks) once the outage is over.

Thanks for the understanding.

[Resolved] Maintenance outage

Wed, 09 Oct 2024 20:00:00 +0200

Database volume needs to be enlarged to accomodate the load on the Packit Service. To minimize the amount of jobs affected by this maintenance there will be no jobs processed, but they should be enqueued until the outage is over.

In case you are affected, please retrigger the jobs accordingly (via comment on the affected pull requests, or Re-Run button on GitHub Checks).

Thanks for understanding.

[Resolved] Fedora infrastructure authentication problem

Mon, 22 Jan 2024 22:00:00 +0100

The Fedora account system is currently down, leading to failures of jobs that interact with it (mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update) as Packit is unable to authenticate. To stay updated on the root cause of this problem, please track this issue.

The problem should be now resolved. You can retrigger the failed jobs by:

/packit propose-downstream comment in any upstream issue
/packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request

[Resolved] Cluster upgrade

Mon, 02 Oct 2023 08:00:00 +0000

A cluster upgrade is scheduled to start on Mon October 2nd at 08:00 UTC and last approximately 90 minutes.

We do not expect any significant disruption.

The started jobs might be executed with a delay and if you think Packit does not react to your events, try to retrigger the action via comment command.

[Resolved] Outage

Tue, 15 Aug 2023 04:00:00 +0200

We are having an outage that started on Tuesday EU early morning. Jobs during this time could get stuck in pending state. This is caused by a failed database schema migration because of a lack of memory during the automatic deployment of the new production images. We are working on the fix and apologize for the inconvenience.

Update: Migration is completed. You might need to retrigger the jobs (i.e. via comment).

[Resolved] Cluster Update

Wed, 03 May 2023 10:00:00 +0200

The OpenShift cluster is currently updating from 4.11.28 to 4.12.13

[Resolved] Issue with pushing to dist-git

Tue, 11 Apr 2023 18:28:00 +0200

Due to a new pre-push hook being applied in dist-git, we are unable to create pull-requests from fork as part of propose-downstream and pull-from-upstream jobs.

We are sorry about that and are working on a way how to resolve this.

[Resolved] SLO1 Error Budget issues

Thu, 09 Mar 2023 10:00:00 +0100

We are having SLO1 error budget issues.

These are mostly caused by performance issues, we are working on them and prioritizing more improvements (see https://github.com/packit/packit-service/issues/1954 ).

We are sorry for any slowness you can experience.

[Resolved] Running grouping migrations

Tue, 14 Feb 2023 02:00:00 +0100

We are updating our database schema to facilitate further development. As a part of this process, multiple database migrations will have to be run. The first step is grouping together related builds and test runs. The migrations will happen during the next deployment to production, on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to take roughly 40 minutes, after which the service should come back up.

Since significant refactoring was paired with these changes, there may be some bugs/regressions that we did not spot on staging. Please, contact us via the usual means if you spot anything suspicious.

[Resolved] SRPM builds stuck in Submitted state

Fri, 20 Jan 2023 21:19:06 +0100

Workers for processing long-running tasks got stuck on Friday evening with to most visible consequence being the SRPM builds stuck in ‘Submitted’ state.

The workers were restarted on Sat morning and all SRPMs were sent to Copr to build and the jobs pipelines were processed.

The cause is not known at the time of resolving the issue.

[Resolved] Outage

Thu, 24 Nov 2022 23:24:00 +0100

We are having resource issues in our cluster and particular parts of the deployment can be down.

Updates:

2022-11-25T08:35:00+01:00: Stage instance has been scaled down so our production instance has more resources. Production instance should be fine now. We are investigating more.

[Resolved] Copr builds and tests on commits and releases ignored

Tue, 01 Nov 2022 03:00:00 +0100

There was a bug introduced with Tuesday’s production update causing Copr builds and tests for commit and release triggers were not executed.

[Resolved] Issues with the task execution

Tue, 11 Oct 2022 13:57:00 +0200

We are experiencing issues with the execution of the tasks. (The statuses are waiting in the The task was accepted. state.) We are investigating the cause. Sorry about the possible troubles. We will try to fix this as soon as possible.

Updates:

Tue Oct 11 12:24:02 UTC: We are slowly going through the queue. Please, be patient.
Tue Oct 11 13:12:36 UTC: We have gone through the queue and you should experience regular response times. The original cause is still not known but the problem probably occured during an SRPM upload to Copr. (It can be a fault on our or Copr side.)
Wed Oct 12 05:27:46 UTC: Service operated normally since the last update. To reduce the chance of this, we plan to move the building of SRPMs to Copr. (Currently default for new installations and opt in for other projects.)

[Resolved] Failing SRPM builds done in Copr

Tue, 23 Aug 2022 04:00:00 +0200

SRPM builds happening in Copr (for packages using the srpm_build_deps config key) were failing due to a dependency issue.

[Resolved] SRPM builds done in Copr are failing

Tue, 21 Jun 2022 10:18:22 +0200

SRPM builds happening in Copr (for packages using the srpm_build_deps config key) are failing due to a dependency issue. We are working on a fix.

[Resolved] Longer response times

Tue, 07 Dec 2021 09:06:14 +0100

Our service got overwhelmed by the events coming from a Fedora dist-git (caused by the epel-9 branching) causing the longer responses.

We have scaled up the workers and the queue is getting shorter. We are planning the next steps to be resilient next time. Sorry about the disturbance.

update: The queue is now empty and you can expect regular operational times.

[Resolved] Moving production Packit Service to a new cluster

Thu, 18 Nov 2021 01:00:00 +0000

We are going to move deployment of our production Packit Service to a new cluster. It should take about 1/2 an hour, during which the workers will be down and queries to API & dashboard won’t work properly because the database will be being migrated. Any requests submitted to the API meanwhile should, however, be fulfilled once the workers are up again.

[Resolved] PostgreSQL db upgrade

Fri, 04 Jun 2021 07:00:00 +0000

Upgrading our PostgreSQL database from version 10 to version 12.