Skip to content

2014

  • 2 min read

Update on scheduled downtime Dec 27-28 and Jan 3-4

Update Jan. 04: The outage is over; all services have been restored.

 Update Jan. 02: We've just migrated most services to the offsite server, and taken the others offline for the second (and last) scheduled outage. We expect to be back online for good Sunday evening.

 Update Dec. 28: Power was restored at 7pm PDT as expected, and all services are now back online. Everything we had planned (powering on the servers remotely via IPMI, copying files/db from the offsite host, etc.) worked great during both the transition to and away from the offsite server. We will do the same thing next weekend. If you still notice any problems, please contact us.

As we found out earlier this month, there will be a power outage in Hearst Gym during the weekend of December 27-28 and January 3-4.

Normally, this would result in all services being completely unavailable. However, we've put in a lot of effort to reduce the impact by transferring as much content to an off-site server as possible. Here's a summary of what to expect:

  • Web hosting will keep working for most accounts. All student group websites have been copied, and almost all individual accounts.

    We copied all individual accounts which have had web traffic in the past month.

    We copied all student groups websites, but student groups with email virtual hosting will not be able to use the offsite server, and will be down during the weekends. Unfortunately, we aren't able to switch A records for these virtual hosts, so there's no way for us to keep these sites available during the downtime while complying with university policy on off-site hosting. Only about 7% of student groups are using email virtual hosting; the rest will be able to use the off-site server.
  • Email hosting and forwarding will be unavailable. There's not much we can do about this, unfortunately. Mails will be delayed by the sending server automatically, and you'll receive them shortly after the outage ends.
  • MySQL will be available on the off-site server. If your website requires MySQL, it will continue to work.
The main OCF website will be available, but the wiki will not. Other services (like SSH, F/OSS mirrors, etc.) will be unavailable.

We've spent a lot of time trying to minimize the impact of the power outage, but there are some things we can't do (we're extremely limited by the university's policies on off-site hosing, and our own lack of resources).

If you have any questions, you can email us at help@ocf.berkeley.edu; we'll be able to view and respond to mail during the outage.
  • 2 min read

Scheduled downtime: Dec 18, Dec 27-28, Jan 3-4

We found out yesterday that, due to construction, Hearst Gym will have no power on Dec 27-28 and Jan 3-4. Unfortunately, all OCF services will be affected by the power outage.

We're looking into ways to reduce the impact, but currently you should expect the following impacts:

  • Web hosting: All web hosting, including student group hosting, will be unavailable. We're working on providing a descriptive error page, rather than simply having requests time out.
  • Email hosting: Email sent to students or to groups with virtually-hosted mail will be delayed until the outage ends. Senders might receive a notice that delivery has been delayed, but you will still receive the messages shortly after the power returns.
These services will be completely unavailable:
  • Database (MySQL) access
  • Shell (SSH/SFTP to tsunami)
  • F/OSS Mirrors (mirrors.ocf.berkeley.edu)
We're working now to try to minimize the impact of the outage, and will post updates here. Please email us if you have any questions.

Update 12/10: We are scheduling downtime during the evening of Thursday, December 18th to test our ability to start all servers and services remotely. Total downtime should be less than 30 minutes.

Update 12/18: Maintenance for tonight is completed. Total downtime was about 45 minutes (instead of the expected 30) due to a problem with a switch after we restored power. The good news is that we caught it now rather than in a week when nobody will be around to fix it. Everything else worked as expected.
  • 1 min read

WordPress XSS vulnerability; please update!

A vulnerability was recently discovered in WordPress which affects a large number of OCF web hosting users. The vulnerability can potentially allow a malicious person to hijack your session and compromise your website.

All users should update immediately to the latest version of WordPress. Versions 3.9.3, 3.8.5, 3.7.5, 4.0.0, 4.0.1 are unaffected by this vulnerability, but we highly advise to always use the latest version.

Updating WordPress is extremely easy; it's just a single click after logging in to the admin panel.

Recent versions of WordPress come with automatic updates enabled for minor releases, which can help to protect you from future vulnerabilities. We strongly recommend not disabling this feature!

If we've contacted you and you need help updating your site, please don't hesitate to get in touch so that we can help!

  • 1 min read

Kernel updates Nov. 08

All OCF servers will be restarted Saturday night (11/08) in order to apply security updates. Downtime should be no more than 15 minutes.
  • 1 min read

Announcing dedicated hosting for web applications!

We're excited to announce a new OCF service for student groups: dedicated hosting for web applications like Rails, Django, Flask, and Node.js!

Previously, hosting for modern web apps was only available via FastCGI, which was difficult to set up and manage. The new service makes it possible to host any app that can bind to a socket, enabling you to run basically any type of application.

OCF hosting for web apps is a pretty cool choice for student groups compared to hosting off site; it's easy to get a berkeley.edu domain name and to get support from friendly volunteer staffers during staff hours, and now it's easy to set up your app on our powerful servers, all hosted on-campus.

We're opening up the new application hosting on a trial basis, and hoping to work closely with a small number of student group early adopters to work out kinks and improve the service. If you're interested, take a look at the documentation and get in touch!

  • 2 min read

Moving www.ocf.berkeley.edu to HTTPS-only

On November 22nd, all websites hosted under the www.ocf.berkeley.edu domain will begin using HTTPS instead of plain HTTP. We will redirect all requests to the corresponding HTTPS site.

Virtually-hosted websites will continue to use plain HTTP, so won't be affected. There also shouldn't be any impact on sites which consist of static content.

We recommend to change all of your absolute links to point to the HTTPS version. You can start doing this immediately; it's already supported! Since we will set up a 301 redirect, no links will be broken.

There are a few things which some sites are currently doing which will cause problems under HTTPS:

  • Including resources from non-secure pages. Some browsers will refuse to load this content, while others will load it but display a degraded security icon.
  • POSTing to non-secure URLs. If your website has forms which post to non-secure URLs, you need to update them to use https instead of http. Although we will set up a redirect, browsers will not necessarily follow this redirect when submitting forms. Additionally, browsers may give a warning if submitting a form which sends data to an insecure URL.
If your website does either of the two thing above, you must fix it before November 22nd, or parts of your website may break. If you use WordPress or another CMS, it is usually sufficient to update the URL in the admin panel.

We appreciate that this may be disruptive to many websites, and have not made the decision lightly. There are many reasons to use HTTPS by default, even for websites which don't contain sensitive information or collect passwords. This is an inevitable transition for the OCF at some point, and while it will be painful, it is easier to make it now than in another four years, when even more sites will be affected.

We estimate that there are about 90 OCF users (~0.3% of OCF users) with active websites which will be affected by the change. We'll try to contact them individually about the change. If you need assistance making changes to your website, don't hesitate to come in during staff hours or send us an email.
  • 1 min read

mirrors.ocf.berkeley.edu - outdated Debian package repos

Today we noticed that our Debian package mirrors had not receive updates since October 2nd, although syncs had been completing successfully. This was due to an issue with mirrors.kernel.org, our upstream mirror, which had not synced in the past week.

Since our syncs were completing normally, we weren't alerted to the problem until today when apt warned us that our mirror was out-of-date. We didn't really consider the possibility that mirrors.kernel.org, which is a top-tier Debian mirror (and one of several which ftp.us.debian.org can resolve to) would receive no updates for an extended period of time. We'll add additional health checks to make sure that not only are syncs completing, but that we're receiving updates.

We contacted the mirrors.kernel.org admins to report the problem, who were very responsive and fixed the problem within 30 minutes. Our mirror is now up-to-date.

  • 1 min read

Online account tools maintenance

OCF's online account tools will be unavailable for a few days while we perform maintenance and upgrades. During this time, requesting an account and resetting passwords via CalNet will not be possible.

There will be no impact on LDAP or other services, and password changes (assuming you know the old password) can be done via the "passwd" command.

Update 2014-08-18: We've completed the maintenance on our online account tools. Note that the URLs have changed; see the wiki or main website for updated links.

  • 1 min read

WordPress xmlrpc exploit

WordPress recently announced a bug in their xmlrpc implementation which can result in denial of service attacks by using large amounts of CPU. Many OCF-hosted sites are running affected versions of WordPress.

In response to activity which took out our webserver for about 15 minutes earlier tonight, we are blocking access to xmlrpc.php files to protect the shared OCF webserver. If you would like to request xmlrpc.php files be unblocked from your site, please contact us.

Now would be a good opportunity to make sure all the software on your website is up-to-date!

  • 1 min read

Campus-wide network issues

As of about 9:00am Tuesday morning, Berkeley campus has been experiencing severe network issues, including high packet loss and latency. Open Computing Facility servers are affected by this outage.

This is a problem on IST's side; updates are available from them.