# Changeset 1365 for docs/HPCA2012/04-methodology.tex

Ignore:
Timestamp:
Aug 24, 2011, 1:55:27 PM (8 years ago)
Message:

Fixed methodology

File:
1 edited

### Legend:

Unmodified
 r1362 \section{Methodology} \section{Evaluation Framework} \label{section:methodology} In this section we describe our methodology for the measurements and investigation of XML parser energy consumption and performance.  In brief, for each of the four XML parsers under study we propose to measure and evaluate the energy consumption required to carry out XML well-formedness checking, under a variety of workloads, and as executed on three different Intel processors. \paragraph{XML Parsers}\label{parsers} To begin our study we propose to first investigate each of the XML parsers in terms of the Performance Monitoring Counter (PMC) hardware events listed in the PMC Hardware Events subsection. Based on the findings of previous work \cite{bellosa2001, bertran2010, bircher2007} we have chosen several key hardware performance events for which the authors indicate a strong correlation with overall performance and energy consumption of the application. In addition, we measure the runtime counts of SIMD instructions and bitwise operations using the Intel Pin binary instrumentation framework. Based on these data we gain further insight into XML parser execution characteristics and compare and constrast each of the Parabix parser versions against the performance of standard industry parsers. The foundational work by Bellosa in \cite{bellosa2001} as well as more recent work in \cite {bircher2007, bertran2010} demonstrate that hardware-usage patterns have a significant impact on the energy consumption characteristics of an application \cite{bellosa2001, bircher2007, bertran2010}. Further, the authors demonstrate a strong correlation between specific PMC events and energy usage. However, each author differs slightly in their opinion of the exact set of PMCs to use. The following subsections describe the XML parsers under study, XML workloads, the hardware architectures, PMC hardware events selected for measurement, and the energy measurement instrumentation set up. We analyze the performance of each of the XML parsers under study based on PMC hardware event counts and contrast their energy consumption measurements based on direct measurements. In our evaluation we evaluate Parabix against two widely available software parsers.  Xerces-C++, and Expat XML parsers. Parabix is our open-sourced XML parser that leverages Parallel Bit Stream technology and the SIMD capabilities of modern commodity processors.  Xerces-C++ version 3.1.1 (SAX) \cite{xerces} is a validating open source XML parser written in C++ available as part of the the Apache project. Expat version 2.0.1 \cite{expat} is a non-validating XML parser library written in C. \subsection{Parsers}\label{parsers} \paragraph{XML Workloads}\label{workloads} XML is used for a variety of purposes ranging from databases to config files in mobile phones. A key feature of these XML files that affects the overall parsing performance is the \textit{Markup density}. \textit{Markup density} is defined as the ratio of the total markup contained within an XML file to the total XML document size.  This metric has substantial influence on the performance of traditional recursive descent XML parser implementations.  We use a mixture of document-oriented and data-oriented XML files in our study to provide workloads with a full spectrum of markup densities. The XML parsing technologies selected for this study are the Parabix1, Parabix2, Xerces-C++, and Expat XML parsers. Parabix1 (parallel bit Streams for XML) is our first generation SIMD and Parallel Bit Stream technology based XML parser \cite{Parabix1}.  Parabix1 leverages the processor built-in {\em bitscan} operation for high-performance XML character scanning as well as the SIMD capabilities of modern commodity processors to achieve high performance.  Parabix2 \cite{parabix2} represents the second generation of the Parabix1 parser. Parabix2 is an open-source XML parser that also leverages Parallel Bit Stream technology and the SIMD capabilities of modern commodity processors. However, Parabix2 differs from Parabix1 in that it employs new parallelization techniques, such as a multiple cursor approach to parallel parsing together with bit stream addition techniques to advance multiple cursors independently and in parallel. Parabix2 delivers dramatic performance improvements over traditional byte-at-a-time parsing technology.  Xerces-C++ version 3.1.1 (SAX) \cite{xerces} is a validating open source XML parser written in C++ by the Apache project.  Expat version 2.0.1 \cite{expat} is a non-validating XML parser library written in C. Table \ref{XMLDocChars} shows the document characteristics of the XML input files selected for this performance study.  The jawiki.xml and dewiki.xml XML files represent document-oriented XML inputs and contain the three-byte and four-byte UTF-8 sequence required for the UTF-8 encoding of Japanese and German characters respectively.  The remaining data files are data-oriented XML documents and consist entirely of single byte $7$-bit encoded ASCII characters. \begin{table*} \end{table*} \subsection{Workloads}\label{workloads} Markup density is defined as the ratio of the total markup contained within an XML file to the total XML document size.  This metric has substantial influence on the performance of traditional recursive descent XML parser implementations.  We use a mixture of document-oriented and data-oriented XML files in our study to provide workloads with a full spectrum of markup densities. \paragraph{Platform Hardware} SSE extensions have been available on commodity Intel processors for over a decade since the Pentium III. They have steadily evolved with improvements in instruction latency, cache interface, and register resources, and the addition domain specific instructions. Here we investigate SIMD extensions across three different generations of intel processors. Table \ref{hwinfo} describes the Intel multicores we investigate. We compare the energy and performance profile of the Parabix under the platforms.  We also analyze the implementation specifics of SIMD extensions under various microarchitecture. We we evalute both the legacy SSE and newer AVX extensions supported by Sandybridge. Table \ref{XMLDocChars} shows the document characteristics of the XML input files selected for this performance study.  The jawiki.xml and dewiki.xml XML files represent document-oriented XML inputs and contain the three-byte and four-byte UTF-8 sequence required for the UTF-8 encoding of Japanese and German characters respectively.  The remaining data files are data-oriented XML documents and consist entirely of single byte $7$-bit encoded ASCII characters. We propose to investigate each the execution profiles of XML parsers using the the Performance Monitoring Counter (PMC) hardware event found in the processor. We have chosen several key hardware performance events which provide insight into the profile of our application and indicate if the processor is doing useful work~\cite{bellosa2001, bertran2010}.  The set of performance counters included in our study are Branch instructions, Branch mispredictions, Integer instructions, SIMD instructions, and Cache misses. In addition, we characterize the SIMD operations and study the type and class of SIMD operations using the Intel Pin binary instrumentation framework. \subsection{Platform Hardware} \paragraph{Intel \CO{}} Intel \CO{} processor, code name Conroe, produced by Intel. Table \ref{core2info} gives the hardware description of the Intel \CO{} machine. \begin{table*}[h] \end{tabular} \caption{Platform Hardware Specs} \label{hwinfo} \end{table*} Intel \CITHREE\ processor, code name Nehalem, produced by Intel. The intent of the selection of this processor is to serve as an example of a low end server processor. Table \ref{i3info} gives the hardware description of the Intel \CITHREE\ machine. Intel \CIFIVE\  processor, code name \SB\, produced by Intel. Table \ref{sandybridgeinfo} gives the hardware description of the Intel \CITHREE\ machine. Each of the hardware events selected relates to performance and energy features associated with one or more hardware units.  For example, total branch mispredictions relate to the branch predictor and branch target buffer capacity. The set of PMC events used included in this study are as follows. Processor Cycles, Branch Instructions, Branch Mispredictions, Integer Instructions, SIMD Instructions and Cache Misses. \subsection{Energy Measurement} We measure energy consumption using the Fluke i410 current clamp applied on the 12V wires that supply power to the processor sockets. The clamp detects the magnetic field created by the flowing current and converts it into voltage levels (1mV per 1A current). The voltage levels are then monitored by an Agilent 34410a multimeter at the granularity of 100 samples per second. This measurement captures the power to the processor package, including cores, caches, Northbridge memory controller, and the quick-path interconnects \cite{clamp}. \paragraph{Energy Measurement} A key benefit of the Parabix parser is its more efficient use of the processor pipeline which reflects in the overall energy usage.  We measure the energy consumption of the processor directly using a current clamp. We apply the Fluke i410 current clamp \cite{clamp} to the 12V wires that supply power to the processor sockets. The clamp detects the magnetic field created by the flowing current and converts it into voltage levels (1mV per 1A current). The voltage levels are then monitored by an Agilent 34410a digital multimeter at the granularity of 100 samples per second. This measurement captures the instantaneous power to the processor package, including cores, caches, northbridge memory controller, and the quick-path interconnects. We obtain samples throughout the entire execution of the program and then calculate overall total energy as  $12V*\sigma^{N_{samples}}_{i=1} Sample_i$.