Mirrors scrub in progress
Mirrors FS scrub in progress, performance may be impacted. ETA 48 hours.
News from the staff team¶
[RESOLVED] Scheduled downtime on 11/13
We are planning on having some scheduled downtime on Saturday 11/13 starting around 12:00pm PST. We expect to be done before 4pm. We will be rebooting our main hypervisor as well as some other machines. Our website, student group websites, and other OCF services will be unavailable for the duration of this downtime.
If you have any questions about the downtime please contact us.
Update 3:57PM 2021-11-13: We have experienced some issues updating our main production server, so we expect to take at least another hour. We apologize for the inconvenience.
Update 7:14PM 2021-11-13: We have been told to leave the build which hosts our servers by the Student Union staff, therefore we likely will not be able to bring things back up until Monday.
Update 4:49AM 2021-11-14: Most services are now up. We are working on bringing scurvy (staff VM, Berkeleytime etc.) up.
Update 5:28AM 2021-11-14: Scurvy is up, berkeleytime is up, hellfire is up, marking this as RESOLVED?
Update 5:40AM 2021-11-14: Marking this as RESOLVED
HTTP/2 is now enabled for web hosting
HTTP/2 is enabled for https connections on the web hosting server (death) as of 1730 PDT, Oct. 23.
Please email us at help [at] ocf.berkeley.edu (or send a message in #rebuild) if you experience any issues.
centos-debuginfo mirror decommissioning
For years, we have been mirroring the centos-debuginfo project, which hosts the *-debuginfo packages for CentOS. However, our centos-debuginfo mirror is an unofficial one, and is not listed on the CentOS website.
Our traffic statistics show that the mirror has no to negligible actual usage, with tens to hundreds of megabytes of data transferred daily (almost entirely by bots).
The project currently occupies 2.7 TiB of disk space, which is the highest of all the projects we mirror, and yet no user is actually using it. As a result, we will decommission the centos-debuginfo mirror on Saturday, October 30, 2021 at around 1200 PDT. The project will continue to be synced in the meantime.
If you are currently using our centos-debuginfo mirror, or if you have any concerns or questions, please email us as soon as possible at help@ocf.berkeley.edu.
We expect to have 3 TiB of usable disk space on the mirrors server following the removal of centos-debuginfo, so if you have any suggestions for open source projects to mirror, please also email us at help@ocf.berkeley.edu :) We are currently discussing adding ubuntu-ports.
rjz
OCF Staff
[Investigating] Mirrors disk status
10/11 0000 PST: We noticed that one disk attribute was abnormal on multiple disks in the mirrors server. All SMART attributes are currently normal, so we currently do not expect any immediate service interruption as we investigate the issue.
10/12: Scheduled maintenance at 1930 PST. ETA 15 mins.
We will update this post once we have more information on the situation.
Emergency Mirrors Outage 10/8
Our mirror server is currently down for maintenance. ETA 60 mins.
Update: 1530 PST This incident has been resolved.
Tanglu, Hyperbola decommissioning
Update: Maintenance was completed during Friday's downtime. No service interruptions expected today. Tanglu and Hyperbola mirrors have been decommissioned.
We are planning to take the mirrors server down for disk array migration from 10am-6pm this Saturday. HTTP, HTTPS, FTP, and Rsync will be unavailable during the maintenance window. The server may come up before 6pm, but please expect degraded performance and/or service interruption throughout the maintenance window.
We are planning to decommission tanglu and tanglu-releases as the project has been defunct for over 2 years. We strongly recommend against the continued use of Tanglu, but if you are still using Tanglu, please change your source.list to another archive mirror. As of Oct 7, 2021, http://ftp-stud.fht-esslingen.de/Mirrors/archive.tanglu.org/tanglu/ is available.
We are also planning to decommission Hyperbola. Our Hyperbola mirror has been out of date for quite a while now (almost 3 years!), but please change your configuration accordingly if you are using Hyperbola and still have us as an entry.
We apologize for the inconvenience.
OCF staff
Emergency Mirrors Maintenance
2021-09-27 19:05 PST: mirrors are back up, this downtime should be resolved. If you notice any issues using our mirrors please reach out to us at help@ocf.berkeley.edu.
We are currently (9/22) running emergency maintenance on mirrors.ocf.berkeley.edu. Due to an earlier drive failure we've had to replace the drives. We expect the mirrors server to be down until Monday, but we will keep this blog post updated.
Intermittent service outages during summer break
The OCF is currently aware of intermittent service outages during the summer break to our OCF website, web hosting services, mailing services, mirrors, and internal infrastructure. Volunteer staffers have begun an investigation of the issue and are planning to resolve it in the coming weeks. Please expect delays in this process due to the limited number of staff available in the summer and other factors beyond our control. We will update this post once we have more information on the situation.
If you are a user or represent a group that has been adversely impacted by a recent OCF service outage, please do not hesitate to contact help@ocf.berkeley.edu so we can investigate further.
Update: This has been resolved.
Major Mirrors Outage
We are suffering from a significant RAID array failure of our mirrors server and thus will not be able to serve any of our mirrors traffic. We expect this down time to last for at least another week while we bring up a new server to serve mirrors.
We hope you can understand that this will take longer than we would hope due to limited site access and reduced staffing over the summer.