Openstack Swift hackathon report

Posted on by

The Openstack Swift community held their mid-cycle meeting last week. It was hosted by SwiftStack in their new San Francisco HQ. It was attended by 16 contributors representing 8 different companies: Red Hat[3], Intel[1], SwiftStack[3], HP[2], IBM[1], NTT[3], Fujitsu[1] and Rackspace[2].

We started the week listing what looked liked an almost impossible large number of topics to be discussed. The main focus of the hackathon was the Erasure Coding work that is on-going and targeted for the Kilo release, but we created a list with over 20 topics to be discussed and worked on in 4 days.

I’m glad to say we were able to cover each one of the topics we listed, resulting in a very productive week. The output of those discussions were either in the form of patches being merged after reviews, new patches being submitted for review or specs being written to record design ideas and decisions for new upcoming features. Projects have been proposed or taken-on by the different members of the community and the companies they represent. Basically, everyone left with many action items for the projects they are working on.

To accomplish our goal of covering all those topics we had to break up into smaller groups and have some of the work be done in parallel, this enabled contributors to participate in the discussions of projects they were already working on or new features they are interested in.

Below is a quick summary/status of the discussions I participated in:

Erasure Codes:

EC is being targeted to be released as part of the Kilo release, during this week we reviewed code regarding the PUT and GET requests as well as the reconstructor. Testing has been limited so far as the GET patches were only posted this week. Up until now it was possible to write data, but not retrieve it. With the majority of the GET code base being put up for review, the amount of testing being done should pick up quickly. This work is being led by Paul Luse at Intel and SwiftStack.
Specs and Patches:

Encryption:

Data at-rest encryption is still in the design phase. There is a spec up for review and a lot of new discussion happened all week. HP and IBM were leading the discussions on this feature.
Specs:

Container Sharding:

One of Swift’s weak points is container performance degradation as the size of containers grow. A spec has been proposed on how to solve this issue with container-sharding. The first version of the spec listed 3 alternatives and during the hackathon one of the alternatives was chosen as the most adequate. The work on this should continue with a new version of the spec being put up for review and then some code patches. This work is being led by Matthew Oliver from Rackspace.
Spec:

Etags on large objects:

Currently Swift calculates the etag header of large objects (SLO) differently from “regular” objects. While the etag of a “regular” object is the md5 checksum of its content. The etag of a large object is the md5 of the concatenation of each one of its segments etag. This is only useful if the client application has that information available and can perform the same calculation. NTT has been looking for ways to calculate the etag of a large object based on the content of the data, just like “regular” objects. This has proven to be a difficult problem to solve and more investigation will be needed.

Storlets (IBM):

During the Paris Summit, IBM presented a couple of talks on a new technology they have been researching called Storlets. This technology would allow the use of Swift storage nodes as compute nodes. Typically data processing is done on compute nodes separate from where data is actually stored. The problem with this approach is that large amounts of data need to be transferred, which becomes inefficient. Storlets would move the processing of the data to storage systems instead of the other way around.

The good news about this effort is that IBM is looking to open source this technology. During the hackathon we held a conference call with IBM to hear a status of the project and discuss ideas on where this new code should live and how it can be integrated with Swift.
Blog on Storlets:

Change partition power:

Another weak point of Swift is that deployers must choose their partition power when first deploying their cluster and can never change it, which makes it very difficult for operators that are looking to start with a small cluster and grow overtime. A spec has been proposed to start new work on solving this problem. An overview was given of the proposed solution and implementation ideas were discussed. Alistair Coles from HP proposed the spec with references to previous work done in this area by Christian Schwede (Red Hat).
Spec:

Fast-Post:

Alistair Coles (HP) has a design spec and couple of patches up for review to solve the issue of changing some specific container metadata on POST. The biggest challenge with this work is making sure that metadata is correctly replicated and reconciled on a eventually-consistent system such as Swift.
Specs and patches:

Change policy on container:

NTT is also investigating ways of allowing Swift admins to change the storage policy on a container. For example, a container is created with a 3-replica policy and after some time the admin would like to change it to a 2-replica policy. NTT proposed some ideas on how the data could be migrated without the need to move the data to a new container. There is no spec for this yet.

Object undelete:

Christian Schwede (Red Hat) led the discussion on some ideas to provide a new feature where operators would be able to configure their clusters to hold on to data for some configurable period of time after a DELETE request. This would help prevent un-wanted data deletion caused by either mistakes or abuse of the system.
Spec:

fsync on dirs:

Prashanth Pai (Red Hat) has a patch up for review regarding fsyncing directories after a rename. This is would close the small window of data loss vulnerability but at a cost of performance loss. Ideas were discussed on how to best address the issue.
Patch:

Object Versioning middleware:

We had a live code review of this patch that I have been working on. Currently object versioning is embedded in the PUT method of the proxy and this patch moves that code to a new middleware. Some needed changes were identified and a new patch will be put up for review soon.
Patch:

Single Process:

I have been leading this project, which will allow for better performance when deploying storage policies with third-party backends. We had a walk through of the current code that is up for review for this project. The idea has been well received by the community and core developers have demonstrated willingness to review and work together with us. During the week we identified common places where a refactoring of the Object controller will benefit both this project as well as the EC project.
Patch:

Container aliases:

The basic idea is to set an alias to a container, for example with a target container inside another account. This would make it much easier to access that container, because the user doesn’t need to use a different storage URL for access. A while ago Christian Schwede (Red Hat) wrote a middleware and proposed this (https://review.openstack.org/#/c/62494), but due to some edge cases we weren’t able to merge it yet. The biggest problem is the eventual consistency when setting the alias. The discussion came up again, thus a spec has been submitted for review. Hopefully we will be able to find a consensus on this and how to proceed.
Spec:

Community:

There were some discussions on how to better provide status of on-going projects to the community at large. We also had some talks on the spec review process and bug list grooming.

There were also talks and code reviews on a number of other projects: Service Tokens, ring optimizations, patches on replicator and updater, Container tempurl keys, FQDN in rings, and Keystone policy (ACLs) files.

John Dickinson (Swift PTL) has also written a summary of this event here.

P.S.: It’s important to note that these projects do not represent all the work that is happening on the Swift community, rather it’s just a subset of the projects that the people attending the Hackathon are working on.

Leave a comment