OpenStack backed with Ceph Erasure Coded Pools

If you’re impatient, skip to the solution section

Over the last few months I’ve been working with the University of Cape Town on the Ilifu research cloud project. The focus for the initial release of the cloud is mainly to provide compute and storage to astronomy and bioinformatics use cases.

The technology powering this cloud is the ever-growing-in-popularity combination of OpenStack (Queens release) as the virtualisation platform and Ceph (Luminous) as the storage backend. We’re utilising the Kolla and Kolla-ansible projects to deploy the OpenStack side of things. I am the lead on the Ceph deployment and opted for the Ceph-ansible method of deployment.

We ran into some issues getting the OpenStack services to work on the Ceph cluster when using erasure coded pools…

More …

Using CWL and Toil to Wrap an Ad-hoc Astronomy Data Processing Pipeline

Blasting Students with Science
Blasting Students with Science

I was recently invited to give a workshop on reproducible scientific workflows to students as part of the Inter-university Institute for Data Intensive Astronomy’s (IDIA) “JEDI” programme. The overall purpose of this workshop was to introduce students from the African continent to various topics that are being dealt with in the data science space. A large focus here was machine learning.

This post details some of my experiences with preparing the original pipeline, working CWL around it and also teaching people how to do it.

More …

DNS Woes With NGINX Reverse Proxy

While configuring some of the internal services that we host for external access through our NGINX proxy VM, I started noticing some strange behaviour. Every once in a little while, when requesting a page that was being passed through the proxy, the proxy server would respond with a 502 Bad Gateway. It turns out that there were some issues with the resolver module for NGINX. I’ll detail how I fixed it below.

More …

Generating Let's Encrypt Wildcard Certificates

With the recent release of Let’s Encrypt’s ACMEv2 protocol implementation, they’ve gained the ability to not only supply SSL certificates for single domains, but also all subdomains. I’ve been interested in switching from our previous CA to Let’s Encrypt when their wildcard support dropped, because it makes renewal of certificates significantly easier due to automation capabilities of the platform. This blog post describes how to generate a wildcard certificate using Certbot.

More …

Installing Ubuntu Server 16.04 on Hardware Backed Software RAID1 (FakeRAID)

I encountered an issue booting Ubuntu Server 16.04 when installing it on a SuperMicro server that had been configured with Software RAID 1. The operating system install script would detect that a RAID environment was active and it would install correctly, but when booting into the newly installed system I would be presented with a blank screen and a blinking cursor. Grub wasn’t even loading.

To solve this issue I booted into a live CD of Ubuntu 16.04 and did the following from the terminal:

More …