The OCF is currently investigating an ongoing service outage that has disrupted web services including our main website (ocfweb) and user-hosted websites. All other services, such as our login servers (tsunami), HPC, and mirrors, should be functioning as normal.
Due to the timing of this outage, our response remains limited throughout the early night. We will update this post once we have more information on the situation.
We are replacing a drive in our mirrors server so it is down. This is expected to be done by 2PM PST 3/10/20.
In addition, from 12:01AM on 05/10, we will temporarily shut down the mirrors service to allow the RAID array to rebuild properly. This may cause a prolonged outage.
EDIT: The rebuild is now complete. We're syncing the mirrors live so you might see some outdated packages.
We're observing extended issues in syncing a number of our mirrors including Ubuntu and Kali, and so some of them may be very out of date.
We're currently working on resolving these issues and will update when we have more information.
Update: the issue has been resolved
The current situation with COVID has caused us to fall behind on a number of software upgrades that are critical to keeping our systems running efficiently and securely, especially on our physical servers. Fortunately we've been able to secure a block of time on from the university in order to perform these upgrades. However, in order to upgrade our physical hypervisors it may be necessary to shut down some production virtual machines. We will try to limit downtime to as little as possible.
The following services (and possibly others) may be intermittently down during this time period (9/20/20 12-5pm):
The following services will be down for an extended period of time:
We suggest you do not schedule large jobs on HPC and have plans in place if your site is expecting a large number of visitors during this time period.
As part of our work to transition from stretch to buster for our MySQL server, we'll be migrating user data today around 9:34pm.
To do this, we'll be setting up replication on the primary onto a dev instance, which we will then switch the 'mysql' hostname over to. There will be a short cutover period once we switch over the host. This post will be updated if any issues arise.
EDIT: The upgrade has been completed. We were under 10 minutes of read-only time and around 5 minutes of downtime.
As of today, RSO signatory status for the previous school year has expired. This means if you were a signatory previously, you will no longer be able to create a new OCF group account for it or reset the password to an existing one using the web interface.
We encourage you to re-register your RSO and add signatories as soon as possible, and are evaluating options depending on how long it takes for signatory status to be reinstated in CalLink especially with the turbulent current state of affairs. Most likely, we'll be forced to suspend account creation until we can verify your status as a signatory. If you need help resetting a password to an existing OCF group account contact help@ocf.berkeley.edu.
The OCF is currently undergoing an unplanned reboot of our main hypervisor which may disrupt most if not all of our services. We expect to be back up shortly.
Edit (19:08 PDT): Most services are back, but we're still working to stabilize all services and make sure services like HPC are fully working.
Edit (19:22 PDT): All services should be back. Let us know at help@ocf.berkeley.edu if you notice that something is down.
The OCF is currently investigating a service outage that has disrupted various services such as our main website (ocfweb), ircbot, Keycloak, and more. All other services, such as web hosting and mirrors, should be functioning as normal.
We will update this post once we have more information on the situation.
Update 01:03 PM: Updated the scope of the outage to include ocfweb.
Update 01:21 PM: The issue has been traced back to internal DNS issues and has now been resolved. We will continue to monitor our services for issues.
Many OCF services are currently unavailable, including the OCF website and personal/group web hosting services.
We are investigating the issue and will update this status post as we learn more.
Update 10:10 AM: We have resolved the root problem and are gradually bringing services back up
Update 10:48 AM: We believe all services are operational. We are continuing to monitor. If you're having issues, please send us an email at help@ocf.berkeley.edu.
The OCF will be closed until further notice in the interest of public health.
Due to the large amount of traffic OCF gets during our opening hours, we have decided it would be best to close the lab to prevent possible spread of the coronavirus.
We apologize for any inconvenience that this may cause, and if you are in urgent need of technology resources or printing, the following places provide computers and printers for use:
- Unit 1, 2, and 3 Academic Center
- Foothill Academic Center
- Clark Kerr Academic Center
- Moffitt Library
- Doe Library
- Jean Gray Hargrove Music Library
- Art History/Classics Library
- Environmental Design Library
- Physics Library
- and more
OCF services besides the physical lab space, including webhosting and mail, are expected to remain available throughout the closure.
We will constantly evaluate the situation and reopen when it improves.
Thanks for Flying OCF!
- The OCF Staff