Unified Quality-of-Service and Data-Lifecycle Definitions for Data Storage and Access - P6 BOF session

25 September 2015- BREAKOUT 7 - 09:30

Users of any storage system will have expectations on how that storage system will behave when they interact with it. Beyond simply accepting and later providing access to their data, these users expect the likelihood of data loss is sufficiently low and the system is sufficiently agile. They may have other expectations, such as the locality of their data. Satisfying these expectations often involves trade-offs between different characteristics (e.g., available capacity vs likelihood of data-loss) which is only possible if the storage system knows these expectations. Such expectations are not expressible through the standard POSIX API. Many systems provide a mechanism for users to describe their expectations. However, such mechanisms are bespoke and often limited.

Collectively, the user's expectations form the Quality-of-Service (QoS) that a storage system is to provide. In some cases, the QoS is formally agreed in some Memorandum of Understanding, Service Level Agreement, or similar document. However, in most cases, the QoS is defined more nebulously.

Over time, the desired QoS of data may change. There may be various motivations for a changing QoS; for example, removing an embargo (so data becomes public), data becoming less interesting (so may be stored on slower media), old and uninteresting data cannot be stored indefinitely (so may be deleted). Such changes may be known in advance and, collectively, form the Data-Lifecycle (DL) and are often triggered by the passing of time.

We use the term Software Defined Storage (SDS) to represent the ability of users to describe the desired QoS and DL to the storage system, either during data ingest or later on. The SDS concept allows more direct engagement between the system responsible for storing data and those users of that data. It may allow users to inspect whether the QoS is being upheld. More importantly, it allows users to adjust the desired QoS (based on changing demand) and to delegate responsibility for handling DL transitions to the storage system.

To facilitate the wide-spread adoption of SDS, there needs to be a common, natural language for the SDS concepts. The simple terms of this language need to be defined (e.g., what does "retention policy" mean?). There also needs to be a grammar for combining these terms to define the user's expected QoS and DL. The long-term goal is to define an API that allows a user to interact with storage systems, based on this grammar, to adjust how the storage system is storing data or how the storage system will react automatically to future events. 


The goal of this meeting is to start a group that is tasked to collect existing QoS and DL practise from existing communities and to distil this into a document that defines a language for describing QoS and DL. This document will be used in follow-on activity that will define and implement an SDS interface to support QoS and DL transitions. 


Plenary 6 (Sep 23-25, Paris)

- Introduction round

- Identification of stakeholders / those present with knowledge of QoS

and DL use-cases.

- Collection of known QoS definitions:

These may be written as "I expect that ..."; for example:

"I expect that three copies exist for each file.",

"I expect that two copies are located in different racks.",

"I expect that opening a file takes less than 100 ms.".

- Collection of known DL definitions:

These may be written as "<time-period>, <QoS>"; for example:

"For the first month, I expect that the data is stored in fast media",

"After the first month, I expect that the data is stored in slower media",

"After 6 months, I expect that the data is made public",

"After 5 years, I expect that the data is deleted unless it is referenced in a paper".

- Definition of teams, individuals to describe the collected definitions

Plenary 7 (Spring 2016)

- Introduction round

- Identification of stakeholders / present and missing ones for QoS and DL,

- Review of QoS definitions.

- Review of DL definitions.

- Discussion of a recommendation

- Definition of one team to form a recommendation

Plenary 8 (Autumn 2016)

- Introduction round

- Identification of stakeholders / present and missing ones

- Review of recommendation

- Finalisation of recommendation

We anticipate work concluding with Plenary 8.

Contact: paul.millar@desy.de