commit 3e01a9b341ca07282b6211d22c54152bdce982f2
Author: Marcos Assuncao <assuncao@acm.org>
Date:   Sat Aug 23 16:33:32 2014 +0200

    Reorganisation of paper content

diff --git a/papers/2014/kwapi/paper.tex b/papers/2014/kwapi/paper.tex
index 7c153f4..4d3cbc5 100755
--- a/papers/2014/kwapi/paper.tex
+++ b/papers/2014/kwapi/paper.tex
@@ -121,11 +121,39 @@ In our work we draw lessons from previous approaches to monitor and analyse ener
 \section{The \ac{KWAPI} Architecture}
 \label{sec:architecture}
 
-Depending on the number of monitored devices and the frequency at which measurements are taken, wattmeters can generate a large amount of data, which requires storage capacity for further processing and analysis. Although management systems often store and perform pre-processing locally on monitored nodes, such an approach poses a few challenges when measuring power consumption as: it can impact on the CPU utilisation and hence influence the power that nodes consume; and depending on the power management policy in place, resource managers can switch off idle nodes or set them to stand by mode to save energy. Centralised storage, on the other hand, allows for faster data access and processing, but can generate more traffic given that measurements need to be continuously transferred over the network to a central point. Once stored, components like OpenStack's Ceilometer can easily retrieve the data.
+An overview of the \ac{KWAPI} architecture presented in Figure~\ref{fig:architecture}. The architecture follows a publish/subscribe model based on a set of layers comprising:
 
-Wattmeters available in the market vary in terms of physical interconnection, communication protocols, packaging and precision of measurements they take. They are mostly packaged in multiple outlet power strips called \acp{PDU} or \acp{ePDU}, and more recently in the \ac{IPMI} cards embedded in the computers themselves. In our work, we used \ac{IPMI} initially to shutdown and turn on compute nodes, but nowadays we also use it to query a computer chassis remotely.
+\begin{itemize} 
+\item \textbf{Drivers}, considered data producers responsible for measuring the power consumption of monitored resources and providing the collected data to consumers via a communication bus; and 
+\item \textbf{Data Consumers} --- or \textbf{Consumers} for short --- that subscribe to receive and process the measurement information. 
+\end{itemize}
+
+\begin{figure*}[!htb]
+\center
+\includegraphics[width=0.7\linewidth]{figs/architecture.pdf}
+\caption{Overview of \ac{KWAPI}'s architecture.}
+\label{fig:architecture}
+\end{figure*}
+
+The communication between the layers is handled by a bus, as explained in detail later. Data consumers can subscribe to receive information collected by drivers from multiple sites. Both drivers and consumers are easily extensible to support, respectively, several types of wattmeters and provide additional data processing services. A REST API is designed as a data consumer to provide a programming interface for developers and system administrators. In this work is used to interface with OpenStack by providing the information (\textit{i.e.} by polling monitored devices) required by a \textit{\ac{KWAPI} Pollster} to feed Ceilometer.
+
+The following sections provide more details about the main components of the KWAPI architecture and their relationship with OpenStack Ceilometer.
+
+\subsection{Driver Layer}
+
+Drivers are threads initialised by a Driver Manager with a set of parameters loaded from a file compliant with the OpenStack configuration format. These parameters are used to query the meters (\textit{e.g.} IP address and port) and determine the sensor ID to be used in the collected metrics. The measurements that a driver obtains are represented as \ac{JSON} dictionaries that maintain a small footprint and that can be easily parsed. The size of dictionaries varies depending on the number of fields set by drivers (\textit{i.e.} whether message signing is enabled). 
+
+Figure~\ref{fig:json} shows a simple example of a \ac{JSON} payload containing one measurement. Optional fields such as voltage and current can be included. ACK messages have a fixed size of 66 bytes when using TCP connection; drivers and data consumers communicate via IPC sockets when running on the same machine.
 
-% the central agent and a dedicated pollster we developed
+\begin{figure}
+\includeJSON{figs/measurement.json}
+\caption{Example of \ac{JSON} payload.}
+\label{fig:json}
+\end{figure}
+
+Drivers can manage incidents themselves, but the manager also checks periodically if all threads are active, restarting them if necessary. It is important to avoid losing measurements because the reported information is in W instead of kWh. The loss of a measurement may be significant. 
+
+Wattmeters available in the market vary in terms of physical interconnection, communication protocols, packaging and precision of measurements they take. They are mostly packaged in multiple outlet power strips called \acp{PDU} or \acp{ePDU}, and more recently in the \ac{IPMI} cards embedded in the computers themselves. Support for several types of wattmeter has been implemented, which drivers can use to interface with a wide range of equipments. In our work, we used \ac{IPMI} initially at Nova to shutdown and turn on compute nodes, but nowadays we also use it to query a computer chassis remotely.
 
 Although Ethernet is generally used to transport \ac{IPMI} or SNMP packets over IP, USB and RS-232 serial links are also common. Wattmeters that use Ethernet are generally connected to an administration network (isolated from the data centre main data network). Moreover, wattmeters may differ in the manner they operate; some equipments send measurements to a management node on a regularly basis (push mode), whereas others respond to queries (pull mode). Other characteristics that differ across wattmeters include: 
 
@@ -135,9 +163,7 @@ Although Ethernet is generally used to transport \ac{IPMI} or SNMP packets over
 \item methodology applied to each measurement (\textit{e.g.} mean of several measurements, instantaneous values, and exponential moving averages).
 \end{itemize}
 
-To illustrate the differences among wattmeters, Table \ref{tab:wattmeters} shows the characteristics of equipments we deployed and evaluated in our data centres.
-
-The granularity at which measurements are taken is another important factor as the needs often vary depending on what one wishes to evaluate. Taking measurements at one-second interval or smaller is common under several scenarios. This can be a challenge in an infrastructure comprising hundreds or thousands of nodes, demanding efficient and scalable mechanisms for transferring information on power consumption.
+Table \ref{tab:wattmeters} shows the characteristics of equipments we deployed and evaluated in our data centres.
 
 \begin{table}
 \centering
@@ -165,54 +191,18 @@ ZEZ LMG450     & Serial                    & 0.05 & 0.01 \\
 \end{footnotesize}
 \end{table}
 
-Furthermore, leveraging the capabilities offered by existing cloud platforms like OpenStack, can help the adoption of a monitoring system, ease deployment, and reduce its learning curve. In addition, users and systems administrators need management reports and visualisation tools to analyse the impact of energy management policies and quickly retrieve relevant data for further analysis.  
-
-\begin{figure*}[!htb]
-\center
-\includegraphics[width=0.9\linewidth]{figs/architecture.pdf}
-\caption{Overview of \ac{KWAPI}'s architecture.}
-\label{fig:architecture}
-\end{figure*}
-
-Hence, we summarise the main requirements for the energy monitoring platform as follows: 
-
-\begin{enumerate}
-\item \label{req:data_storage}\textbf{Reliable data storage}: a centralised storage where energy consumption data can be placed and easily retrieved. Centralised storage here does not imply data is stored on a single node. Systems like Apache Hadoop HDFS\footnote{http://hadoop.apache.org/}, Apache Cassandra\footnote{http://cassandra.apache.org/}, and MongoDB\footnote{http://www.mongodb.org/} could be used.
-
-\item \label{req:heterogeneous_meters}\textbf{Handle heterogeneous wattmeters}: there is a need for handling multiple device types and to design the architecture in a way that support for new wattmeters can be easily included.
-
-\item \label{req:efficient_communication}\textbf{Efficient communication}: the envisioned system should provide means for nodes to efficiently communicate their energy consumption to components interested in processing it. A message bus could be used to manage this communication efficiently.
-
-\item \label{req:open_source}\textbf{Integration with open source cloud platform}: the proposed system should interface with existing open source cloud platforms to ease deployment and use.
-
-\item \label{req:visualisation}\textbf{Visualisation and reports}: the system should offer a set of management reports that provide quick feedback to administrators and users during execution of tasks and applications. In addition, it should provide means and APIs for advanced queries.
-\end{enumerate}
-
-The following sections describe the \ac{KWAPI} architecture and how it addresses the aforementioned requirements.
-
-\subsection{\ac{KWAPI}}
-
-Figure~\ref{fig:architecture} depicts the \ac{KWAPI} architecture, which is based on a set of layers comprising drivers, responsible for performing measurements, and plug-ins that subscribe to receive the collected information. The communication between the two layers is handled by a bus as explained later. Following a publish/subscribe architecture, plug-ins can subscribe to receive information collected by drivers from multiple sites. Drivers and plug-ins are easily extensible to support other types of wattmeters, and provide additional services. The API is designed to provide a programming interface for developers and system administrators, and is used to interface with OpenStack by providing the information (\textit{i.e.} by polling monitored devices) required to feed Ceilometer.
-
-Ceilometer's central agent and a dedicated pollster that we developed are used to publish energy metrics. They query the API plug-in and publish cumulative (kWh) and gauge (W) counters. These counters are not associated with a particular tenant, since a server can host multiple clients simultaneously. By exploring Ceilometer in our framework, we address two requirements, namely (\ref{req:data_storage}) by leveraging the data repository and storage abstraction layers that Ceilometer provides; and (\ref{req:open_source}) via integration with OpenStack. The following subsections provide more details on the framework layers.
 
-\subsubsection{Driver Layer}
+\subsection{Data Consumers}
 
-Drivers are threads initialised by a manager with a set of parameters loaded from a file compliant with the OpenStack configuration format. These parameters are used to query the meters (\textit{e.g.} IP address and port) and determine the sensor ID to be used in the collected metrics. The measurements that a driver obtains are represented as \ac{JSON} dictionaries, which have the advantage of being human readable and easily parsed, while keeping a small footprint. The size of the dictionaries may vary depending on the number of fields set by the drivers (\textit{i.e.} whether message signing is enabled). Figure~\ref{fig:json} shows an example of a \ac{JSON} payload containing one measurement. Optional fields can be added, such as voltage and current. ACK messages have a fixed size of 66 bytes (on a TCP connection). When drivers and API are on the same machine, they communicate via IPC sockets.
+A data consumer retrieves and processes measurements taken by drivers and provided via bus. Consumers expose the information to other services including Ceilometer and visualisation tools. By using a system of prefixes, consumers can subscribe to all producers or a subset of them. When receiving a message, a consumer verifies the signature, extracts the content and processes the data. By default \ac{KWAPI} provides two data consumers, namely the REST API (used to interface with Ceilometer) and a visualisation consumer.
 
-\begin{figure}
-\includeJSON{figs/measurement.json}
-\caption{Example of \ac{JSON} payload.}
-\label{fig:json}
-\end{figure}
-
-As incidents may occur, the manager periodically checks if all threads are active, restarting them if necessary. Drivers can manage incidents themselves, but if for any reason they stop their execution, they are automatically restarted by the manager. It is important to avoid losing measurements because the information reported is in W and not kWh; the loss of a measurement is hence significant. 
+\subsubsection{REST API}
 
-Moreover, support for several types of wattmeter has been implemented, which drivers can use to interface with a wide range of equipments. The concept of drivers and support for multiple wattmeters addresses requirement (\ref{req:heterogeneous_meters}).
+The API consumer computes the number of kWh of each driver probe, adds a timestamp, and stores the last value in watts. If a driver has not provided measurements for a long time, the corresponding data is removed. The REST API allows an external system to retrieve the name of probes, measurements in W or kWh, and timestamps. The API is secured by OpenStack Keystone tokens\footnote{http://keystone.openstack.org}, whereby the consumer needs to ensure the validity of a token before sending a response to the system. 
 
-\subsubsection{Plug-in Layer}
+\subsubsection{Visualisation}
 
-A plug-in retrieves and processes measurements taken by drivers and provided via bus. Plug-ins expose the information to other services including Ceilometer and visualisation tools. By using a system of prefixes, plug-ins can subscribe to all sensors, a subset, or to other plug-ins. After verifying a message signature, they extract its content and process the data. By default, \ac{KWAPI} provides two plug-ins, namely an API to interface with Ceilometer, and a visualisation tool.
+The visualisation consumer builds \ac{RRD} files from received measurements, and generates graphs that show the energy consumption over a given period, with additional information such as average electricity consumption, minimum and maximum watt values, last value, total energy and cost in Euros. \ac{RRD} files are of fixed size and store several collections of metrics with different granularities. A web interface displays the generated graphics and a cache mechanism triggers the creation of graphs during queries only if they are out of date. These visualisation resources offer quick feedback to administrators and users during execution of tasks and applications. Figure~\ref{fig:graph_example} shows an example of generated graph. 
 
 \begin{figure*}[!htb]
 \center
@@ -221,17 +211,23 @@ A plug-in retrieves and processes measurements taken by drivers and provided via
 \label{fig:graph_example}
 \end{figure*}
 
-\begin{itemize}
+\subsection{Internal Communication Bus}
 
-\item \textbf{API for Ceilometer}: the API plug-in computes the number of kWh of each probe, adds a timestamp, and stores the last value in watts. As mentioned earlier, this data is stored by Ceilometer using its own database. If a probe has not provided measurements for a long time, the corresponding data is removed. This plug-in has a REST API that allows a client to retrieve the name of the probes, measurements in W or kWh, and timestamps. The API is secured by using OpenStack Keystone tokens\footnote{http://keystone.openstack.org}, whereby a plug-in needs to ensure the validity of a token before sending a response to a client. 
-  
-\item \textbf{Visualisation}: the visualisation plug-in builds \ac{RRD} files from received measurements, and generates graphs that show the energy consumption over a given period, with additional information (average electricity consumption, minimum and maximum watt values, last value, total energy and cost in Euros). \ac{RRD} files are of fixed size, and store several collections of metrics with different granularities. Figure~\ref{fig:graph_example} shows an example of generated graph. In addition, a web interface displays the generated graphics and a cache mechanism triggers the creation of graphs during queries only if they are out of date. This plug-in is implemented to satisfy requirement (\ref{req:visualisation}).
-\end{itemize}
+\ac{KWAPI} uses ZeroMQ \cite{HintjensZeroMQ:2013}, a fast broker-less messaging framework written in C++, where transmitters play the role of buffers. ZeroMQ supports a wide range of bus modes, including cross-thread communication, IPC, and TCP. Switching from one mode to another is straightforward. ZeroMQ also provides several design patterns such as publish/subscribe and request/response. As mentioner earlier, in our publish/subscribe architecture drivers are publishers, and data consumers are subscribers. If no data consumer is subscribed to receive data from a given driver, the latter will not send any information through the network.
+
+Moreover, one or more optional forwarders can be installed between drivers and data consumers to minimise network usage. Forwarders are introduced . Forwarders are designed to act as especial data consumers who subscribe to receive information from a driver and multicast it to all normal data consumers subscribed to receive the information. Forwarders enable the design of complex topologies and optimisation of network usage when handling data from multiple sites. They can also be used to bypass network isolation problems and perform load balancing.
+
+\subsection{Interface with Ceilometer}
+
+We opted for integrating KWAPI with an existing open source cloud platform to ease deployment and use. Leveraging the capabilities offered OpenStack can help in the adoption of a monitoring system and reduce its learning curve.
+
+Ceilometer's central agent and a dedicated pollster (\textit{i.e.} \ac{KWAPI} Pollster) are used to publish and store energy metrics into Ceilometer's database. They query the REST API data consumer and publish cumulative (kWh) and gauge (W) counters that are not associated with a particular tenant, since a server can host multiple clients simultaneously. 
 
+Depending on the number of monitored devices and the frequency at which measurements are taken, wattmeters can generate a large amount of data thus demanding storage capacity for further processing and analysis. Management systems often store and perform pre-processing locally on monitored nodes, but such an approach can impact on CPU utilisation and influence the power consumption. In addition, resource managers may switch off idle nodes or set them to stand by mode to save energy, which make them unavailable for processing. Centralised storage, on the other hand, allows for faster data access and processing, but can generate more traffic given that measurements need to be continuously transferred over the network to a central point. Hence, a centralised storage where energy consumption data can be placed and easily retrieved. 
 
-\subsubsection{Internal communication bus}
+Ceilometer using its own database, which is used here to store the energy consumption metrics. In this way, systems that interface with OpenStack's Ceilometer, including Nova, can easily retrieve the data. It is important to notice that, even though Ceilometer provides the abstraction of a central repository for metrics, it also uses a database abstraction that enables the use of distributed systems such as Apache Hadoop HDFS\footnote{http://hadoop.apache.org/}, Apache Cassandra\footnote{http://cassandra.apache.org/}, and MongoDB\footnote{http://www.mongodb.org/}. 
 
-\ac{KWAPI} uses ZeroMQ \cite{HintjensZeroMQ:2013}, a fast broker-less messaging framework written in C++, where transmitters play the role of buffers. ZeroMQ supports a wide range of bus modes, including cross-thread communication, IPC, and TCP. Switching from one mode to another is straightforward. ZeroMQ also provides several design patterns such as publish/subscribe, and request/response. In our publish/subscribe architecture drivers are publishers, and plug-ins are subscribers. One or more forwarders between them broadcast packets to all plug-ins subscribed to receive information from a given probe. Thanks to forwarders, the network usage is optimised because the packets a driver generates are sent only once regardless of the number of plug-ins registered to receive it. If a probe is not listened by any plug-in, its measurements are not sent to forwarder and hence not transmitted over the network. The forwarders not only reduce dramatically the network usage, but allow one to build flexible architectures, by bypassing network isolation problems, or performing load balancing. Although ZeroMQ and forwarders have been employed to address requirement \ref{req:efficient_communication} (\textit{i.e.} efficient communication), in the next section we evaluate the scalability and limitations of a bus to transfer power-consumption measurements in large data centres. 
+The granularity at which measurements are taken and metrics are computed is another important factor because user needs vary depending on what they wish to evaluate. Taking measurements at one-second interval or smaller is common under several scenarios, which can be a challenge in an infrastructure comprising hundreds or thousands of nodes, demanding efficient and scalable mechanisms for transferring information on power consumption. Hence, in the next section we evaluate the throughput of KWAPI under a few scenarios.
 
 
 \section{Performance Evaluation}
@@ -298,7 +294,7 @@ Although the CPU usage often depends on the drivers, plug-ins, and their complex
 
 %While measuring the network usage, our experiments showed a transfer rate of around 230KB/s with message signing enabled and around 135KBs/s otherwise. Message signing overall introduces an overhead of 70\%. Sending large packets can be explored to decrease the packet overhead. If several drivers send measurements simultaneously, ZeroMQ provides an optimisation mechanism that aggregates the data into a single TCP datagram. Figure~\ref{fig:packet_size} shows the number of packets under the evaluated scenarios. We noticed that certain packets contain up to forty measurements.
 
-Although a measurement interval of one second meets the requirements of users in our platform, we wanted to evaluate the impact of using a communication bus in the transfer of observations between drivers and API. In a second experiment we used two machines. On the first machine we instantiated 1,000 driver threads placing random observations on the communication bus. On the second machine we measured the number of measurements that the API is able to receive over a minute. We varied the time between measurements from 0.2 to 1.0 seconds. Figure~\ref{fig:measurement_intervals} summarises the obtained results. Though the number of observations generated in this experiment is much higher than what we currently need to handle in our platform, we observe that the framework is able to transfer measurements from drivers to API under a 0.4 second interval without adding much jitter. Under smaller measurement intervals, however, observations start to accumulate and are transferred at large chunks. We believe that under small measurement intervals, and consequently a very large number of observations per second, an architecture based on stream processing systems that guarantee data processing might be more appropriate. Regarding our requirement (\ref{req:efficient_communication}), although the framework suits the purposes of large range of users, if measurements are to be taking at very small time intervals, a stream processing architecture would probably yield better performance.
+Although a measurement interval of one second meets the requirements of users in our platform, we wanted to evaluate the impact of using a communication bus in the transfer of observations between drivers and API. In a second experiment we used two machines. On the first machine we instantiated 1,000 driver threads placing random observations on the communication bus. On the second machine we measured the number of measurements that the API is able to receive over a minute. We varied the time between measurements from 0.2 to 1.0 seconds. Figure~\ref{fig:measurement_intervals} summarises the obtained results. Though the number of observations generated in this experiment is much higher than what we currently need to handle in our platform, we observe that the framework is able to transfer measurements from drivers to API under a 0.4 second interval without adding much jitter. Under smaller measurement intervals, however, observations start to accumulate and are transferred at large chunks. We believe that under small measurement intervals, and consequently a very large number of observations per second, an architecture based on stream processing systems that guarantee data processing might be more appropriate. Although the framework suits the purposes of large range of users, if measurements are to be taking at very small time intervals, a stream processing architecture would probably yield better performance by placing elements to preprocess data closer to where it is generated.
 
 As mentioned earlier, plug-ins can subscribe and select probes from which they want to receive information. If multiple plug-ins select a node, information from the node is sent only once through the network. The architecture also allows for a hierarchy of plug-ins to be established, where a plug-in can be deployed on a site to summarise or compute average values that are placed on the bus to be consumed by higher level plug-ins. 
 
