Skip to content

2015

  • 1 min read

Service outage due to hanging server reboot (resolved)

Public-facing services are down. Web hosting and SSH access have been down since approximately 2am PDT. The login server (tsunami) was restarted at 10:21pm.

Staff are investigating the outage.

Update 3:13am: Services are now recovering and should be fully restored within a few minutes.

Update 3:18am: All services should now be restored. Please contact help@ocf.berkeley.edu if you are still encountering trouble.

  • 1 min read

Lab closed, some services affected by campus-wide network downtime

There was a fire in the campus datacenter a few hours ago which took down CalNet and related campus websites. See systemstatus.berkeley.edu for more details.

 The following were affected by the CalNet outage, but are no longer affected

  • Account creation
  • Password resets


WiFi access in the lab is still down. All OCF services themselves are fine.

 The Publications Center made the decision to close the lab due to the WiFi outage. It will reopen once WiFi has been restored.

  • 1 min read

Downtime Wednesday for security updates

All servers will be restarted the night of Wednesday, August 12th to apply security updates. Sorry for the inconvenience.

 Update: The downtime took longer than expected when our file server failed to come back up within the ~5 minutes we expect for a restart. It took an additional ~30 minutes to run a fsck. We'll be more careful to watch for and schedule these at more convenient times in the future.
  • 1 min read

Ongoing downtime due to server crash

At 1:07pm today, hal, our primary production server, froze. We are on-site and working to restore it. We are moving the important servers to another machine while we investigate, as hal continues to experience issues.

Update 2:07pm: Service has been restored, but we are continuing to move servers to a different machine. There will be some downtime as we continue the migration, but it will affect single services only.

Remaining to migrate: (updated 5:04pm)

  • firestorm (ldap)
  • death (www)
  • pestilence (dns, dhcp)
  • supernova (admin)
  • maelstrom (mysql)
  • tsunami (ssh)
  • anthrax (smtp)
  • sandstorm (group smtp)
  • biohazard (apphost)
  • lightning (puppet)
  • earthquake (accounts)
  • typhoon (rt)
  • blight (wiki)
  • flood (irc)
  • reaper (jenkins)
  • dev-earthquake (dev-accounts)
  • pollution (cups)

 Update 5:20pm: All VMs are migrated to jaws, and all services should be restored. We'll be debugging and rebuilding hal in the near future, and will be scheduling downtime some time in the next few weeks to move VMs back. We'll post a followup here when we have a date in mind.
  • 1 min read

Server maintenance 6/19

We will be updating our physical servers on Friday, June 19th around 9pm PDT. All OCF services will be affected, though we expect downtime to be less than 15 minutes.
  • 1 min read

Directory listings re-enabled by default June 19th

On June 19th, we will re-enable Apache directory listings by default for both virtual hosts and userdir web hosting. You can disable these by creating a file named .htaccess in your web root with the line "Options -Indexes".

  • 1 min read

Online account tools maintenance tonight

The online account tools (used for requesting accounts, changing passwords, etc.) will be offline for about an hour tonight as we rebuild the server. No other services will be affected; you can still change your password via SSH if desired.

  • 1 min read

Mail server maintenance 5/22

Our internal mail server (anthrax) will be unavailable for about an hour tomorrow. This only affects individual accounts with email forwarding, and mail sent from the OCF. Virtual hosts with email forwarding will still work.


No mail will be lost; instead, it will be queued until the server is available again. At the worst, your mail may be delayed by an hour or two.

Staff alumni with Google Apps accounts won't be affected at all.
  • 1 min read

Directory listings disabled by default May 31th

On May 31th, we will disable Apache directory listings by default for both virtual hosts and userdir web hosting. You can re-enable these by creating a file named .htaccess in your web root with the line "Options +Indexes".