Laboratoire de l'Informatique et du Parallélisme » XLcloud

\documentclass[conference]{IEEEtran}

% Add the compsoc option for Computer Society conferences.

\usepackage{ctable}

\usepackage{cite}

\usepackage[cmex10]{amsmath}

\usepackage{acronym}

\usepackage{graphicx}

\usepackage{multirow}

\usepackage{listings}

\usepackage{color}

\usepackage{xcolor}

\usepackage{balance}

\usepackage{algorithm2e}

\acrodef{KWAPI}{KiloWatt API}

\acrodef{KWRanking}{KiloWatt Ranking}

\acrodef{VPC}{Virtual Private Cloud}

\acrodef{HPC}{High Performance Computing}

\acrodef{GPU}{Graphical Processing Unit}

\colorlet{@punct}{red!60!black}

\definecolor{@delim}{RGB}{20,105,176}

\lstdefinelanguage{json}{

    basicstyle=\footnotesize\ttfamily,

    literate=

     *{\ }{{{\ }}}{1}

      {:}{{{\color{@punct}{:}}}}{1}

      {,}{{{\color{@punct}{,}}}}{1}

      {\{}{{{\color{@delim}{\{}}}}{1}

      {\}}{{{\color{@delim}{\}}}}}{1}

      {[}{{{\color{@delim}{[}}}}{1}

      {]}{{{\color{@delim}{]}}}}{1},

}

\newcommand{\includeJSON}[1]{\lstinputlisting[language=json,firstnumber=1]{#1}}

% correct bad hyphenation here

\hyphenation{op-tical net-works semi-conduc-tor}

\begin{document}

\title{A Resource Reservation System to Improve the Support for HPC Applications in OpenStack}

\author{\IEEEauthorblockN{Fran\c{c}ois Rossigneux, Laurent Lef\`{e}vre, Marcos Dias de Assun\c{c}\~ao}

\IEEEauthorblockA{INRIA Avalon, LIP Laboratory\\

Ecole Normale Sup\'{e}rieure de Lyon\\

University of Lyon, France}

}

\maketitle

\begin{abstract}

A reservation system is important to enable users to plan the execution of their applications and for providers to deliver performance guarantees; key to many \ac{HPC} applications. We have developed a reservation framework called Climate that interfaces with Nova, OpenStack's compute scheduler. Climate manages reservations and their placement on physical hosts taking into account several factors, such as energy efficiency. 

\end{abstract}

\IEEEpeerreviewmaketitle

\section{Introduction}

\acresetall

Cloud computing \cite{ArmbrustCloud:2009} has become an important building block to providing IT resources and services to organisations of all sizes. The workload consolidation that clouds provide by virtualising resources and enabling customers to share the same physical infrastructure brings several advantages such as energy efficiency and higher utilisation. The on-demand business model practiced by most cloud providers permits customers to request resources when needed and pay only for what they use.

Though this consolidated model suits most of today's use cases, certain applications that demand \ac{HPC} are not fully portable to this scenario as they are generally resource intensive and sensitive to performance variations. The press occasionally reports on the use of public clouds to perform \ac{HPC}, but reported cases are often large bag-of-task applications whose execution requires almost Herculean effort \cite{ec2supercomputer:2013}. A large part of \ac{HPC} applications, however, still demand homogeneity among computing nodes and predictable network performance. The means employed by cloud providers to offer customers with high and predictable performance mostly consist in deploying bare-metal resources or using specialised virtual machines placed in groups where high network throughput and low latency can be guaranteed. This model may seem in contrast with traditional cloud use cases as it is expensive and provides little flexibility in terms of workload consolidation and resource elasticity. Attempts to use public clouds or co-locate \ac{HPC} applications on the same physical hardware, however, have proven difficult \cite{OstermannHPCEC2:2010,AniruddhaHPCCloud:2013,YoungeVirtHPC:2011,WangNetPerf:2010}.

Over the past \ac{HPC} users have been tolerant to resource availability as they generally share large clusters, to which exclusive access is made via submitting a job that may wait in queue for a time often longer than the job execution itself. We do not consider that clouds should adopt a similar queuing model, but we believe that a compromise between wait time and on-demand access could be explored for \ac{HPC} in the cloud via resource reservations. Reservations provide means for reliable allocation and allow customers to plan the execution of their applications. Reservation models currently in use in public clouds rely on requesting resources in advance for a long time period (\textit{i.e.} from one to three years) or bidding for virtual machine instances in a spot market.

In this paper, we describe a reservation framework for reserving and provisioning specialised \ac{HPC} resources using a popular open source cloud platform. The proposed solution, implemented as a configurable component of OpenStack\footnote{https://wiki.openstack.org}, provides reservation models that are more sophisticated and flexible than those currently offered by public cloud providers. By using discrete event simulations, we demonstrate that the use of reservations of bare-metal resources can bring benefits for both cloud providers and customers.

% Introduction will likely change a lot, but it is written to serve as a guide.

%Section \ref{sec:related_work} describes a motivating scenario on the use of reservations in the cloud; presents background information on OpenStack; and examines related work on the topics of reservation and provisioning of HPC resources in the cloud.

% ----------------------------------------------------------------------------------------

\section{Background and Related Work}

\label{sec:related_work}

Grid'5000 \cite{Grid5000} --- an experimental platform comprising several sites in France and Luxembourg --- provides its users with single tenant environments by using bare-metal provisioning and allowing resources to be reserved in advance. In order to learn more about the impact of reservations on the use of a data centre, we considered a simple experiment. We used the log of advance reservations generated by the scheduler \cite{CapitOAR:2005} one of Grid'5000 sites (\textit{i.e.} the data centre located in Lyon) and simulated a scheduling under two cases, namely the current scenario where users make reservations to use resources at a time in future; and an extrapolation where all requests from the log were treated as immediate reservations, thus emulating a more elastic cloud-like scenario. Figure~\ref{fig:lyon_case} shows the maximum number of cores required to handle requests over time under both cases. Although this is an extreme scenario and certain users would probably not anticipate their reservations if using a cloud, had they known resources were available during work hours, we believe some users would change their behaviour; particularly those who currently use the infrastructure over weekends or at night.

\begin{figure}[ht]

\centering 

\includegraphics[width=1.\columnwidth]{figs/lyon_use_may_2013.pdf} 

\caption{Maximum number of CPU cores required at the g5k's Lyon site.}

\label{fig:lyon_case}

\end{figure}

Grid'5000 is an experimental platform, but similar bursts of requests during a few hours of the day have also been noticed in other systems whose logs have been extensively studied \cite{FeitelsonPWA:2014}. Although bare-metal or specialised VMs minimise performance penalties for running \ac{HPC} applications on a cloud, providing the elasticity with which other cloud users are familiar with can be prohibitive, and advance reservations may be explored to help minimising these costs. The next section describes some of the current reservation models available for clouds and discusses their limitations.

\subsection{Reservation in Clouds}

The benefits and drawbacks of resource reservations have been extensively studied for various systems, such as clusters of computers \cite{SmithSchedARs:2000, LawsonMultiQueue:2002, MargoReservations:2000, RoblitzARPlacement:2006}, meta-schedulers \cite{SnellARMetaScheduling:2000}, computational grids \cite{ElmrothBrokerCCPE:2009,FarooqlaxityARs:2005,FosterGara:1999}, virtual clusters and virtual infrastructure \cite{ChaseCOD:2003}; and have been applied under multiple scenarios including co-allocation of resources \cite{NettoFlexibleARs:2007}, and improving performance predictability of certain applications \cite{WieczorekARWorkflow:2006}. 

As of writing, Amazon Web Services (AWS)\footnote{http://aws.amazon.com/ec2/} offers cloud services that suit several of today's use cases and providers the richest set of reservation options for virtual machine instances. AWS offers four models of allocating VM instances, namely on-demand, reserved instances, spot instances and dedicated --- the latter are allocated within a \ac{VPC}. Under on-demand use, customers are charged on an hourly basis for the instances they allocate. The performance is not guaranteed as resources are shared among customers and their performance depends on the cloud workload. Reserved instances can be requested at a discount price under the establishment of long term contracts, and provide more guarantees in terms of performance than their on-demand counterparts. To request spot instances, a customer must specify a limit price --- often referred to as bid price --- she is willing to pay for a given instance type. If the bid price surpasses the spot market price, the user receives the requested instances. Existing spot instances can be destroyed if the spot price exceeds a user's bid. Instances created within a \ac{VPC} provide some degree of isolation at the network level, but their physical hosts may contain instances from multiple customers. To improve fault tolerance, it is possible to request dedicated instances at a premium, so that instances will not share the host. Under all the models described here, AWS allows users to request \ac{HPC} instances, optimised for processing, memory use, I/O, and instances with \acp{GPU}.

OpenNebula cloud management tool \cite{OpenNebula:2011} provides reservation and scheduling services by using Haizea \cite{SotomayorLeases:2008}. Haizea supports multiple types of reservations (\textit{e.g.} immediate, advanced and best-effort) and takes into account the time required to prepare and configure the resources occupied by the virtual machines (\textit{e.g.} time to transfer virtual machine image files) so that a reservation can start at the exact time a user required. When used in simulation mode, Haizea offers means to evaluate the scheduling impact of accepting a set of reservations. We found that the  Haizea model --- where a scheduler able to handle reservations was incorporated as a module of a cloud resource manager --- as good starting point to provide reservation and \ac{HPC} support for OpenStack. In the following sections we provide background information on OpenStack's compute management architecture, and how we extend it by providing loosely coupled components to handle resource reservations.

\subsection{OpenStack}

OpenStack is an open source cloud computing platform suitable to both public and private settings. It manages and automates deployment of virtual machines on pools of compute resources, can work with a range of virtualisation technologies and can handle a variety of tenants. OpenStack has gained traction and received support from a wide development community who incorporated several features, including block storage, image library, network provisioning framework, authentication and authorisation, among other services. In this work we focused mainly on the compute management service, called Nova \footnote{https://wiki.openstack.org/wiki/Nova}. 

Nova manages a cloud computing fabric and is responsible for instantiating virtual machines on a pool of available hosts running \textit{nova-compute}. Nova uses its \textit{nova-scheduler} service to determine how to place virtual machine requests. Its API (\textit{i.e. nova-api}) is used by clients to request virtual machine instances. In order to make a request for a VM, a client needs to specify a number of requirements (\textit{e.g.} the flavour), which determine the instance to be created. By default, Nova uses a filter scheduler that initially obtains a list of physical hosts, applies a set of filters on the list to select servers that match the client's criteria, and then ranks the remaining servers according to a pre-configured weighing scheme. Nova comes with a set of pre-configured filters, but the choice of filters and weighing is configurable. Nova can also be configured to accept hints from a client to influence how hosts are filtered and ranked upon a new request.

% ----------------------------------------------------------------------------------------

\begin{figure*}[htb]

\centering 

\includegraphics[width=0.65\linewidth]{figs/architecture.pdf} 

\caption{Architecture of the proposed reservation framework.}

\label{fig:reservation_architecture}

\end{figure*}

\section{Proposed Reservation System}

\label{sec:reservation_system}

The architecture of the reservation framework, termed as Climate, is depicted in Figure~\ref{fig:reservation_architecture}. The architecture aims to provide scheduling support for advance reservations without being intrusive to Nova. It consists of the following main components.

\subsection{Reservation API} 

Climate API enables users and client applications to manage reservations. The following parameters should be specified in order to create a reservation:

\begin{itemize}

\item \textbf{host\_properties}: characteristics of required servers;

\item \textbf{start\_time}: earliest time for starting the reservation;

\item \textbf{end\_time}: latest time for completing the reservation;

\item \textbf{duration}: time duration of the reservation; and

\item \textbf{quantity}: number of servers required;

\end{itemize}

If \textbf{start\_time} and \textbf{end\_time} are not specified, the request is treated as an immediate reservation, thus starting at the current time. The API is provided as a REST service that implements the calls described in Table~\ref{tab:climate_api}. The API relies on two backend services, namely \textbf{Climate Inventory} and \textbf{Climate Scheduler}, for storying information about hosts and managing reservations respectively. Handling a reservation request is performed in two phases. First, the API queries Climate Inventory to discover the available hosts that match the criteria specified by \textbf{host\_properties}. Second, a filtered list of hosts, along with the other request parameters, are given to Climate Scheduler, which then finds a time period over which the reservation can be granted.

The API is not only an interface for tenants. Nova uses it to find available hosts and to determine a set of resources associated with a reservation, and modules that need to query the reservation schedule do so via the API. Moreover, the API uses the same security infrastructure provided by OpenStack, including messages carrying Keystone tokens\footnote{http://docs.openstack.org/developer/keystone/}, which are used to allow a client application to discover the hosts associated with a reservation.

\begin{table}

\caption{Climate REST API calls}

\label{tab:climate_api}

\centering

\begin{tabular}{lll}

\toprule

\textbf{Method} & \textbf{URL} & \textbf{Description}\\

\toprule

GET & /properties/ & Lists resource properties\\

\midrule

POST & /reservations/ & Creates a reservation\\

\midrule

GET & /reservations/ & Lists reservations\\

\midrule

GET & /reservations/\{reservation-id\} & Describes a reservation\\

\midrule

DELETE & /reservations/\{reservation-id\} & Cancels a reservation\\

\bottomrule

\end{tabular}

\end{table}

 \subsection{Climate Inventory} 

Climate Inventory is an RPC service used by the reservation API to discover the hosts that are possible candidates to serve a reservation request. The candidates are servers that both match the host properties specified in the request and are available during the requested time (\textit{i.e.} their \textbf{running\_vm} field in Nova's database is set to 0). NovaClient queries Nova's API and the list of potential candidates is filtered using the \textbf{json\_filter} syntax specified by Nova.

As mentioner beforehand, Climate Inventory uses \textbf{host\_properties} as filtering criteria. In order to specify the required hosts a user needs to create such a filter. For this end, the \textbf{/properties/} call of the reservation API provides a catalogue of properties used for filtering. By default, the call shows the properties used for filtering based on the list of hosts registered with Nova, but an admin can choose to disable or enable certain a property.

\subsection{Climate Scheduler}

This component manages the reservation schedule and extends Nova's filtering scheduler by providing a set of resource filters and ranking (or weighing) criteria for handling reservation requests, as described as follows.

\subsubsection{Filtering}

Nova filter accepts a scheduling hint --- in our case used to provide a reservation ID created by Climate. When an ID is provided, the filter uses the reservation API, providing an admin Keystone token, to retrieve the list of hosts associated with the reservation. If no reservation ID is given, Nova still uses the reservation API to establish the list of hosts that have been reserved. Only the hosts that are not in this list can be used to serve the request.

\subsubsection{Ranking}

We created two Nova weighers. The first weigher is ignored by reservation requests and ranks machines according to their free time until the next reservation. If handling a request for a non-reserved instance, the weigher tries to place the instance on a host that is available for the longest period. This helps minimise the chance of having to migrate the instance at a later time to vacate its host for a reservation.

The second weigher, termed as \ac{KWRanking} ranks machines by their power efficiency (\textit{i.e.} FLOPS/Watt). This weigher relies on:

\begin{itemize}

\item A software infrastructure called \ac{KWAPI}, described in detail in our previous work, built for monitoring the power consumed by resources of a data centre and for interfacing with Ceilometer to provide power consumption data. Ceilometer is OpenStack's telemetry infrastructure used to monitor performance metrics\footnote{https://wiki.openstack.org/wiki/Ceilometer}.

\item A benchmark executed on the machines to determine their delivered performance by watt.

\end{itemize}

The goal of this weigher is to prioritise the use of the most power efficient machines, and create windows during which the least efficient resources could be powered off or placed in low power consumption modes. Climate provides an API that enables switching resources on/off, or putting them into standby mode. Choosing between placing a resource in standby mode or switching it off depends on the length of time during which it is expected to remain idle. As switching a resource back on to serve an impending request often takes time, means to estimate future workload are generally important.

% ----------------------------------------------------------------------------------------

\section{Reservation Strategies}

\label{sec:strategies}

To benefit from adding support for reservation of bare-metal resources, we discuss strategies that a private cloud could implement towards reducing the amount of energy consumed by compute resources. These strategies could be used to complement the ranking system described above by, for instance, selecting the least power efficient resources to be switched off.

\subsection{Power-off Idle Resources}

The strategy considered here consist of checking periodically what resources are idle. Once determined that a resource has remained idle for a number of consecutive intervals and it is not committed to serve a reservation over a give time horizon --- \textit{i.e.} when reservation is enabled --- the resource is powered off. Previous work \cite{OrgerieSaveWatts:2008} has evaluated the impact of decisions on appropriate intervals for measuring idleness, for deciding on the horizon for switching off resources committed to reservations. This work considers that the measurement interval, idleness time, and reservation horizon are respectively 1, 5 and 15 minutes. Algorithm~\ref{algo:alloc_policy} summarises the strategy.

\IncMargin{-0.6em}

\RestyleAlgo{ruled}\LinesNumbered

\begin{algorithm}[ht]

\caption{Sample resource allocation policy.}

\label{algo:alloc_policy} 

\DontPrintSemicolon

\SetAlgoLined

\SetAlgoVlined

\footnotesize{

\label{algo:check_idle_start}\textbf{procedure} checkIdleNodes()\;

\Begin{ 

	$Res\_idle_t \leftarrow $ get list of idle resources at interval $t$\;

	$Res\_idle_{t-1} \leftarrow $ get list of idle resources at interval $t-1$\;

	\If{$success = $ \textbf{false}}{

		$success \leftarrow enqueueReq(r)$\;

	}

	\ForEach{resource $r \in Res\_idle_t$}{

	      \eIf{$r \in Res\_idle_{t-1}$}{

	          // increase number of idle intervals of $r$\;

	          $r.idle_intervals \leftarrow r.idle_intervals + 1$\;

	      }{

	          $r.idle_intervals \leftarrow 1$\;\label{algo:check_idle_end}

	      }

	}

}

\BlankLine

\label{algo:switch_start}\textbf{procedure} switchResourcesOnOff()\;

\Begin{ 

	$Res\_on_{t} \leftarrow $ list of resources switched on\;

	$Res\_off_{t} \leftarrow $ list of resources switched off\;

	$Res\_reserv_{t,h} \leftarrow $ resources reserved until horizon $h$\;

	$nres\_reserv_{t,h} \leftarrow $ number of resources in $Res\_reserved_{t,h}$\;

	$nres\_fcast_{t+1} \leftarrow $ forecast number of resources required at $t+1$ \;

	$nres\_req_{t+1} \leftarrow max(nres\_fcast_{t+1},nres\_reserv_{t,h})$\;

	\While{$nres\_req_{t+1} < sizeof(Res\_on_{t})$} {

		$r \leftarrow $ pop resource from $Res_{off}$\;

		switch resource $r$ on\;

		add $r$ to $Res\_on_{t}$\;

	}

	$Res\_idle_t \leftarrow $ get list of idle resources at interval $t$\;

	\While{$nres\_req_{t+1} > sizeof(Res\_on_{t})$} {

	    \ForEach{resource $r \in Res\_idle_t$}{

	    	\If{$r.idle\_intervals \geq 5$ and $r \notin Res\_reserv_{t,h}$}{

		    remove $r$ from $Res\_on_{t}$\;

		    switch resource $r$ off\;

		    add $r$ to $Res\_off_{t}$\;

		}

		\If{$nres\_req_{t+1} == sizeof(Res\_on_{t})$}{

		   \textbf{break}\; \label{algo:switch_end}

		}

	    }

	}

}

}

\While{system is running} {

   every minute call checkIdleNodes()\;

   every 5 minutes call switchResourcesOnOff()\;

}

\end{algorithm}

\IncMargin{0.6em} 

Lines~\ref{algo:check_idle_start} to \ref{algo:check_idle_end} contains the pseudo-code to identify idle resources, whereas lines \ref{algo:switch_start} to \ref{algo:switch_end} determines the resources that need to be switched on or off. 

% ----------------------------------------------------------------------------------------

\section{Experimental Setup and Results}

\label{sec:experiments}

In this section, we evaluate the potential for energy savings when employing a reservation framework in private clouds. We discuss the experimental setup and metrics, and then analyse the obtained performance results.

\subsection{Experimental Setup}

A discrete-event simulator developed in house is used to model and simulate the resource allocation and request scheduling in a private cloud setting. We resort to simulation as it enables controlled, repeatable and large-scale experiments. Both infrastructure capacity and resource requests are expressed in number of CPU cores. As traces of cloud workloads are very difficult to obtain, we use request logs gathered from Grid'5000 sites and adapt them to model cloud users' resource demands. Under normal operation, Grid'5000 enables resource reservations, but users' requests are conditioned by the available resources. For instance, a user willing to allocate resources for an experiment will often check a site's agenda, see what resources are available and will eventually make a reservation during a convenient time frame. If the user cannot find enough resources, she will either adapt her requirements to resource availability --- \textit{e.g.} change the number of required resources, and reservation start or/and finish time --- or choose another site with available capacity. The request traces, however, do not capture what users' initial requirements were before they make their requests.

In order to obtain a workload trace on provisioning of bare-metal resources that is more cloud oriented, we adapt the request traces and infrastructure capacity of Grid'5000 by making the following changes to reservation requests:

\begin{enumerate}

\item \label{enum:cond1} Requests whose original submission time is within working hours and start time lies outside these hours are considered as on-demand requests starting at their original submission time.

\item \label{enum:cond2} Remaining requests are considered as on-demand requests both submitted and starting at their original start time.

\item \label{enum:capacity} The resource capacity of a site is modified to the maximum number of CPU cores required to honour all requests, plus a safety factor.

\end{enumerate}

The characteristics of best-effort requests are not changed. Change \ref{enum:cond1} modifies the behaviour of users who today explore resources during off-peak periods, whereas \ref{enum:cond2} alters the current practice of planning experiments in advance and reserving resources before they are taken by other users. Although the changes may seem extreme at first, they allow us to evaluate what we consider to be our \textit{worst case scenario} where reservation is not enabled. Moreover, as mentioned earlier, we believe the model adopted by existing clouds, where short-term advance reservations are generally not allowed and prices of on-demand instances do not vary over time, users would have little incentives to explore off-peak periods or plan their demand in advance. Change \ref{enum:capacity} reflects the industry practice of provisioning resources to handle peak demand and including a margin of safety.

\subsection{Performance Metrics}

To evaluate the potential for energy savings by introducing support for resource reservation, we first take into account the worst case scenario and quantify the time resources remain idle (\textit{i.e.} $res_{idle}$) from the start of an evaluation $t_{start}$ through its end $t_{end}$, which is given by: 

\[ res_{idle} = \int_{t_{start}}^{t_{end}} ct_{t} - cu_{t} \,dt \]

\noindent where $ct_{t}$ and $cu_{t}$ are respectively the site CPU core capacity and number of cores in use at time $t$. As we consider that computer nodes would ideally be switched off when idle, $res_{idle}$ is taken as upper bound to potential savings. The actual energy savings $e_{savings}$ is the amount of time cores are switched off during interval $t_{start}$ to $t_{end}$ --- \textit{i.e.}, $e_{savings} = \int_{t_{start}}^{t_{end}} c_{off} \,dt$, where $c_{off}$ is the number of switched-off cores).

Switching resources off, however, may lead to a scenario where powered-off resources must be switched back on to serve a request that arrives. Booting up resources takes time and increases the time required to make resources available to users, specially under immediate reservations. Therefore, we assess the impact of switching resources on the quality of service users perceive by computing the aggregate weighted delay $delay_{req}$ of affected requests $R_{delay}$, which is given by:

\[ delay_{req} = \sum_{r \in R_{delay}} \frac{{cores_r} \times {time\_boot_r}}{duration_r} \]

\noindent where $cores_r$ is the number of cores required by request $r$, $time\_boot_r$ is the number of seconds taken by the last allocated resource to boot, and $duration_r$ is the duration of $r$ in seconds.

\subsection{Evaluation Results}

% ----------------------------------------------------------------------------------------

\section{Conclusion}

\label{sec:conclusion}

This work discussed the need for reservation support on cloud resource management. It introduced a framework for OpenStack for enabling reservation of resources, with a focus on bare-metal provisioning for certain high performance computing applications.

% ----------------------------------------------------------------------------------------

\section*{Acknowledgment}

This research is supported by the French Fonds national pour la Soci\'{e}t\'{e} Num\'{e}rique (FSN) XLcloud project. Some experiments presented in this paper were carried out using the Grid'5000 experimental testbed, being developed under the Inria ALADDIN development action with support from CNRS, RENATER and several Universities as well as other funding bodies (see https://www.grid5000.fr). 

\bibliographystyle{IEEEtran}

%\balance

\bibliography{references}

\end{document}