Laboratoire de l'Informatique et du Parallélisme » XLcloud

% \author{\IEEEauthorblockN{Fran\c{c}ois Rossigneux, Marcos Dias de Assun\c{c}\~ao, Laurent Lef\`{e}vre}

% \IEEEauthorblockA{INRIA Avalon, LIP Laboratory\\

% Ecole Normale Sup\'{e}rieure de Lyon\\

% University of Lyon, France}

% }

\item A software infrastructure called \ac{KWAPI}, described in detail in our previous work \cite{Rossigneux:2014}, built for monitoring the power consumed by resources of a data centre and for interfacing with Ceilometer to provide power consumption data. Ceilometer is OpenStack's telemetry infrastructure used to monitor performance metrics\footnote{https://wiki.openstack.org/wiki/Ceilometer}.

As illustrated by Algorithm~\ref{algo:alloc_policy}, the strategy considered here checks periodically what resources are idle (Lines \ref{algo:check_idle_start} to \ref{algo:check_idle_end}). Also periodically, the strategy determines that resources have remained idle over a given time period and whether they will likely remain idle over a time horizon because they are not assigned to any request (Lines \ref{algo:switch_start} to \ref{algo:switch_start}). Once determined that a resource has remained idle for a number of consecutive intervals and it is not committed to serve a reservation over a give time horizon --- \textit{i.e.} when reservation is enabled --- the resource is powered off. Previous work \cite{OrgerieSaveWatts:2008} has evaluated the impact of decisions on appropriate intervals for measuring idleness, for deciding on the horizon for switching off resources committed to reservations. Here we consider that the measurement interval, idleness time, and reservation horizon are respectively 1, 5 and 50 minutes.

\caption{Na\"ive resource allocation strategy.}

	$Res\_idle \leftarrow $ list of idle resources during past $x$ intervals\;

	$Res\_idle_{t} \leftarrow $ list of idle resources at interval $t$\;

	$Res\_idle \leftarrow Res\_idle \cap Res\_idle_{t}$ \label{algo:check_idle_end}

	$Res\_idle \leftarrow $ list of idle resources during past $x$ intervals\;

	// Obtain list of resources that will remain idle during $h$\;

	$Switch\_off = Res\_idle \cap Res\_off_{t} \cap Res\_reserv_{t,h}$\;

  $switchOff(Switch\_off)$\; \label{algo:switch_end}

}

\BlankLine

\label{algo:res_arrival_start}\textbf{procedure} requestSubmitted(Request $req$)\;

\Begin{ 

  $av\_boot\_time \leftarrow $ get average machine deployment time\;

  $now \leftarrow $ get current time\;

  // Find a place for $req$ in the scheduling queue\;

  $schedule(req)$\;

  \eIf{$res.start\_time < now + av\_boot\_time$}{

      $Req\_res \leftarrow $ get list of resources required by $req$\;

      $Res\_off \leftarrow $ list of resources switched off\;

      $Switch\_on \leftarrow Req\_res \cap Res\_off$\;

      $switchOn(Switch\_on)$\;  

   }{  

      $time\_check \leftarrow res.start\_time - av\_boot\_time$\;

      // schedule event to check whether machines need to be\;

      // switched on at time $time\_check$\;

      $check\_machines(req, time\_check)$\; \label{algo:res_arrival_end}

   }

\BlankLine

}

When a request arrives, the strategy finds a place for it in the scheduling agenda. Once the schedule for the request is determined, the strategy verifies whether resources need to be switched on, or if a future resource check must be scheduled (Lines \ref{algo:res_arrival_start} to \ref{algo:res_arrival_end}). This policy is simple and efficient from an energy consumption perspective, but it can lead to high performance degradation if resources need to be constantly switched on or off. 

We believe that users of a cloud infrastructure would plan their resource demands in advance and use reservations if enough incentives were provided. These incentives could materialise in the form of discount prices for allocating resources or information on how their behavioural changes affects resource allocation and maximise energy savings \cite{OrgerieSaveWatts:2008}. In this work we do not focus on devising the proper incentives for users to adhere to using reservations. As discussed in Section \ref{sec:experiments}, the experiments consider that at least part of users find enough incentives to change their allocation decisions and hence reserve resources in advance.

\section{Reservation Workloads and Bare-Metal Deployment}

\label{sec:workloads}

This section describes the workloads used to evaluate the use of advance reservation for provisioning of bare-metal resources in a cloud environment, and the model used for deployment. 

\subsection{Reservation Workloads}

As traces of cloud workloads are very difficult to obtain, we use request logs gathered from Grid'5000 sites and adapt them to model cloud users' resource demands. There are essentially two types of requests that users of Grid'5000 can make, namely \textit{best-effort} and \textit{reservations}. Scheduling of best-effort requests is done as in most batch-scheduling systems, where resources are assigned to requests whenever they become available. Reservations allow users to request a set of resources over a defined time frame. Resource reservations have priority over best-effort requests, which means that the latter are cancelled whenever resources are reserved. In this work we ignore best-effort requests.

Under normal operation, although Grid'5000 enables resource reservations, user requests are conditioned by the available resources. For instance, a user willing to allocate resources for an experiment will often check a site's agenda, see what resources are available and will eventually make a reservation during a convenient time frame. If the user cannot find enough resources, she will either adapt her requirements to resource availability --- \textit{e.g.} change the number of required resources, and reservation start or/and finish time --- or choose another site with available capacity. The request traces, however, do not capture what users' initial requirements were before they make their requests.

Change \ref{enum:cond1} modifies the behaviour of users who today explore resources during off-peak periods, whereas \ref{enum:cond2} alters the current practice of planning experiments in advance and reserving resources before they are taken by other users. Although the changes may seem extreme at first, they allow us to evaluate what we consider to be our \textit{worst case scenario} where reservation is not enabled. Moreover, as mentioned earlier, we believe the model adopted by existing clouds, where short-term advance reservations are generally not allowed and prices of on-demand instances do not vary over time, users would have little incentives to explore off-peak periods or plan their demand in advance. Change \ref{enum:capacity} reflects the industry practice of provisioning resources to handle peak demand and including a margin of safety.

\subsection{Bare-Metal Deployment}

As the provisioning of bare-metal resources by clouds is a relatively new topic, it is also difficult to find workload traces that provide information on the time of operations required to deploy resources, such as switching resources on, cloning operating system images, and partitioning physical disks.

To model the time required for deployment, we use traces gathered from Grid'5000 sites and generated by Kadeploy3 \cite{JeanvoineKadeploy3:2013}. Kadeploy3 is a disk imaging and cloning tool that takes a file containing the operating system to deploy (\textit{i.e} an environment) and copies it to target nodes. An environment deployment by Kadeploy3 consists of essentially three phases:

\begin{enumerate}

 \item Minimal environment setup, where nodes reboot into a minimal environment containing tools required for partinioning disks during the phase.

 \item Environment installation, when the environment is broadcast and copied to all nodes, and post-copy operations are made. 

 \item Reboot of nodes using the deployed environment.

\end{enumerate}

In order to build a model for bare-metal deployment we gathered several years of Kadeploy3 traces from three Grid'5000 sites --- Rennes, Reims and Lyon --- and evaluated the time to execute the three phases described above. Table~\ref{tab:kadeploy_clusters} lists the characteristics of the clusters considered in this study.

\begin{table}[!hbt]

\caption{Clusters whose deployment logs were considered.}

\label{tab:kadeploy_clusters} 

\footnotesize

  \begin{tabular}{ccccl}

    \toprule

    \multirow{2}{10mm}{\centering{\textbf{Cluster Name}}} &

    \multirow{2}{7mm}{\centering{\textbf{Site Name}}} &

    \multirow{2}{7mm}{\centering{\textbf{\# Nodes}}} &

    \multirow{2}{10mm}{\centering{\textbf{Install Date}}} & 

    \multirow{2}{30mm}{\centering{\textbf{Node Characteristics}}}\\

    & & & & \\

    \toprule

    parapluie & Rennes & 40 & Oct. 2010 & \multirow{3}{30mm}{2 CPUs AMD\@1.7GHz, 12 cores/CPU, 47GB RAM, 232GB DISK}\\ 

    & & & \\ 

    & & & \\ \midrule

    parapide & Rennes & 25 & Nov. 2011 & \multirow{3}{30mm}{2 CPUs Intel\@2.93GHz, 4 cores/CPU, 23GB RAM, 465GB DISK}\\ 

    & & & \\ 

    & & & \\ \midrule

    paradent & Rennes & 64 & Feb. 2009 & \multirow{3}{30mm}{2 CPUs Intel\@2.5GHz, 4 cores/CPU, 31GB RAM, 298GB DISK}\\ 

    & & & \\ 

    & & & \\ \midrule

    stremi & Reims & 44 & Jan. 2011 & \multirow{3}{30mm}{2 CPUs AMD\@1.7GHz, 12 cores/CPU, 47GB RAM, 232GB DISK}\\

    & & & \\ 

    & & & \\ \midrule

    sagittaire & Lyon & 79 & Jul. 2007 & \multirow{3}{30mm}{2 CPUs AMD\@2.4GHz, 1 core/CPU, 1GB RAM, 68GB DISK}\\ 

    & & & \\ 

    & & & \\ 

    \bottomrule

  \end{tabular} 

\end{table}

All deployments from Jan. 2010 through Dec. 2013 have been considered. The first step towards building a model for deployment consisted in building time histograms and visually examining what probability distributions were more likely to fit the obtained data. Once this first step were a number of distributions were considered, we found that log-normal, gamma and generalised gamma were more likely to fit the data. Figure~\ref{fig:deploy_fitting} depics the results of fitting the three distributions to the deployment time information of each cluster.

\begin{figure*}

\centering{

\parbox{.32\linewidth}{

\includegraphics[width=1.\linewidth]{figs/deployment_parapluie.pdf} 

}

\begin{minipage}{.32\linewidth}

\includegraphics[width=1.\linewidth]{figs/deployment_parapide.pdf}

\end{minipage}

\begin{minipage}{.32\linewidth}

\includegraphics[width=1.\linewidth]{figs/deployment_paradent.pdf}

\end{minipage}

}

\centering{

\begin{minipage}{.32\linewidth}

\includegraphics[width=1.\linewidth]{figs/deployment_stremi.pdf}

\end{minipage}

\begin{minipage}{.32\linewidth}

\includegraphics[width=1.\linewidth]{figs/deployment_sagittaire.pdf}

\end{minipage}

}

\caption{Deployment time histograms and distribution fitting.}

\label{fig:deploy_fitting}

\end{figure*}

The goodness of fit of the distributions has also been submitted to the Kolmogorov–Smirnov test (KS test), whose statistic (D-statistic) quantifies the distance between the distribution function of empirical values and the cumulative distribution function of the reference distribution. Table~\ref{tab:ks_test} summarises the results for the considered clusters. 

\begin{table}[!hbt]

\centering

\caption{Kolmogorov-Smirnov test for goodness of fit.}

\label{tab:ks_test} 

\footnotesize

  \begin{tabular}{cccc}

    \toprule

    \multirow{2}{15mm}{\centering{\textbf{Cluster Name}}} & \multicolumn{3}{c}{\textbf{D-Statistics}}\\

    \cmidrule(r){2-4}

     & \textbf{Log-normal} & \textbf{Gamma} & \textbf{Gen. Gamma}\\

    \toprule

    parapluie & 0.054 & 0.056 & 0.057 \\ \midrule 

    parapide & 0.088 & 0.089 & 0.087 \\ \midrule

    paradent & 0.039 & 0.036 & 0.037 \\ \midrule

    stremi & 0.053 & 0.056 & 0.055 \\ \midrule

    sagittaire & 0.070 & 0.073 & 0.075 \\

    \bottomrule

  \end{tabular} 

\end{table}

Although the D-statistics does not present great differences among the results obtained by the distributions, log-normal provides slighly better fit to most clusters. We hence choose to base deployment time on a log-normal distribution.

% ----------------------------------------------------------------------------------------

\section{Experimental Setup and Results}

\label{sec:experiments}

In this section, we evaluate the potential for energy savings when employing a reservation framework for bare-metal provisioning in private clouds. We discuss the experimental setup and metrics, and then analyse the obtained performance results.

\subsection{Experimental Setup}

A discrete-event simulator developed in house was used to model and simulate the resource allocation and request scheduling in a private cloud setting. We resorted to simulation as it enables controlled, repeatable and large-scale experiments. We model and simulate two private cloud environments. Both infrastructure capacity and resource requests are expressed in number of machines.

To model the load of the two considered environments, we collected traces of reservation requests from two Grid'5000 sites, namely Lyon and Reims spanning six months, from Jan. 2014 to Jun. 2014. For each environment we consider two scenarios, one where the workload trace is modified according to the rules described in Section~\ref{sec:workloads}; and another using the original trace. The first is termed as a \textit{cloud} scenario, whereas the second is \textit{reservation}. In this way, there is a total of four scenarios --- \textit{lyon\_cloud}, \textit{lyon\_reservation}, \textit{reims\_cloud} and \textit{reims\_reservation}. As mentioned earlier, under cloud scenarios all requests are made by users on demand.

The number of resources of each environment is achieved by simulating their corresponding cloud scenarios under a large number of resources, so that no request is rejected. The maximum number of resources used during the evaluation is taken as the site capacity. In the scenarios we consider here, Lyon and Reims have 195 and 136 machines respectively.

Based on the deployment information from Kadeploy, we model the time required to boot powered-off machines requested by a reservation using a log-normal distribution whose scale is $\log(500)$ and shape is $0.4$.

To evaluate the potential for energy savings by introducing support for resource reservation, we first take into account the worst case scenario and quantify the time resources remain idle (\textit{i.e.} $res_{idle}$) from the start of an evaluation $t_{start}$ through its end $t_{end}$ --- the time of submission of the last request --- which is given by: 

\noindent where $ct$ and $cu$ are respectively the site machine capacity and number of machines in use at a time $t$. As we consider that computer nodes would ideally be switched off when idle, $res_{idle}$ is taken as upper bound to potential savings. The actual energy savings $e_{savings}$ is the amount of time machines are in fact switched off during interval $t_{start}$ to $t_{end}$ --- \textit{i.e.}, $e_{savings} = \int_{t_{start}}^{t_{end}} c_{off} \,dt$, where $c_{off}$ is the number of powered-off machines.

Switching resources off, however, may lead to a scenario where they must be switched back on to serve a request that arrives. Booting up resources takes time and increases the time required to make resources available to users, specially under immediate reservations. Therefore, we assess the impact of switching resources off on the quality of service users perceive by computing the aggregate delay $delay_{req}$ of affected requests $R_{delay}$, which is given by:

% \[ delay_{req} = \sum_{r \in R_{delay}} {machines_r} \times \frac{{time\_boot_r}}{duration_r} \]

\[ delay_{req} = \sum_{r \in R_{delay}} (r_{dep\_start} + r_{dep\_end}) - r_{start\_time} \]

\noindent where $r_{dep\_start}$ is the time at which deployment of the requested nodes started, $r_{dep\_end}$ is when the last machine became ready to be used, and $_{start\_time}$ is the time when the request was supposed to start. As discussed earlier, to model the machine deployment time, we used information collected from Kadeploy3 traces.

Figure \ref{fig:potential_savings} summarises the results for potential energy savings. As the scheduling strategy switches resources off almost immediately once it identifies that they have remained idle for a period and will continue to be in a time horizon, it is able to explore almost all potential savings under both cloud and reservation scenarios. This simple policy does not consider a high cost of powering off/on resources such as issues related to air-conditioning and power supply. Even though the strategy is simple, the cloud and reservation scenarios present different results regarding quality of service.

\begin{figure}[htb]

\centering 

\includegraphics[width=0.95\linewidth]{figs/potential_saving.pdf} 

\caption{Potential savings and aggregate off periods.}

\label{fig:potential_savings}

\end{figure}

As shown in Figure~\ref{fig:request_delay}, the request delay is substantially reduced under scenarios with resource reservations. That is caused because with reservations, resources can be switched on before the reservation starts. Hence, the system does not spend reserved-resource time for environment deployment. 

\begin{figure}[htb]

\centering 

\includegraphics[width=0.95\linewidth]{figs/request_delay.pdf} 

\caption{Aggregate request delay in resource/hour.}

\label{fig:request_delay}

\end{figure}

This work discussed the need for reservation support in cloud resource management. It introduced an OpenStack framework for enabling resource reservation, with a focus on bare-metal provisioning for certain high performance computing applications.