\documentclass[10pt]{article}
\usepackage{fullpage}


\title{MacPorts Statistics - Google Summer of Code 2011}
\author{ Derek Ingrouville - derek@macports.org }

\date{\today}

\begin{document}

\maketitle

\setlength{\parskip}{0.3cm}

\section{Client Side - MacPorts Base}

\subsection{Install}

In order to automatically submit data at regular intervals some small changes had to be made to the installation process. These changes include installing a script which handles data submissions \texttt{submitstats.sh}, configuring \texttt{launchd} to regularly run \texttt{submitstats.sh}, and generating a unique identifier for the user submitting data.

\subsubsection{Makefile.in}
\begin{itemize}
  \item Install \texttt{submitstats.sh} to \texttt{\$(DESTDIR)\${datadir}/macports/}
  \item Run \texttt{setupstats.sh}
\end{itemize}

\subsubsection{configure.ac}

Generate a universally unique identifier to identify this MacPorts installation. The UUID is generated by \texttt{uuidgen} and stored in the variable \texttt{STATS\_UUID}

\subsubsection{Scripts}
\begin{itemize}
\item \texttt{setupstats.sh}

This script is responsible for generating and installing the file \texttt{org.stats.macports.plist}. This plist is used by \texttt{launchd} to regularly run \texttt{submitstats.sh}.

The script takes two arguments
\begin{enumerate}
  \item The path to the script that \texttt{launchd} should execute
  \item The path to the MacPorts configuration file \texttt{macports.conf}
\end{enumerate}

It will execute the script once a week. The day of the week, hour and minute are determined as follows: \newline


\textbf{Weekday:} The day of the week is determined by the machine's hardware UUID modulo 7. This is to help ensure that submissions are roughly evenly distributed throughout the week. \newline
\textbf{Hour: } The hour that \texttt{submitstats.sh} was executed. \newline
\textbf{Minute:} The minute that \texttt{submitstats.sh} was executed. \newline

The plist is installed to \texttt{/Library/LaunchAgents/org.macports.stats.plist} and then loaded by \texttt{launchctl}

\item \texttt{submitstats.sh}

This script has two responsibilities
\begin{enumerate}
  \item Check if a user is participating.
  \item Submit data only if the user if participating
\end{enumerate}
  
It takes one parameter, the path to \texttt{macports.conf}.

To determine if a user is participating it checks if the variable \texttt{stats\_participate} is set to \texttt{yes}. If it is, then \texttt{port stats submit} is executed. If the user is not participating then the script exits.

The reason this script exists is to have a lightweight tool to check if a user is participating before running \texttt{port}. This script will be executed once a week for every user, regardless of whether or not they are participating. 

\end{itemize}

\subsection{Configuration}

Added several variables to \texttt{macports.conf.in} and appropriate descriptions to \texttt{macports.conf.5}.

\begin{itemize}
  \item \texttt{stats\_participate} 
  
  This indicates whether or not a user has chosen to opt-in and share their data. Its value is either \texttt{yes} or \texttt{no}
  
  \item \texttt{stats\_url}
  
  This is the url where data should be submitted.
  
  \item \texttt{stats\_id}
  
  This is the UUID used for submissions. It is initially set to value of the autoconf variable \texttt{@STATS\_UUID@}.
\end{itemize}

\subsection{Changes to macports1.0/macports.tcl}
\begin{itemize}
  \item New Globals
  
    Added globals \texttt{stats\_participate}, \texttt{stats\_url}, \texttt{stats\_id} that correspond to configuration options. \newline
    Added deferred global \texttt{gccversion}
  \item \texttt{gcc} version check
  
    Added proc setgccinfo that is called the first time \texttt{gccversion} is read.
\end{itemize}

\subsection{Changes to pextlib1.0/curl.c - CurlPostCmd()}

Added CurlPostCmd function. This takes two Tcl parameters, the post data and the url.

Example usage is 
\begin{verbatim}
  curl post "project=macports" $url
\end{verbatim}

\subsection{The \texttt{port stats} action}
\texttt{port stats} gathers lists of all active and inactive ports as well as relevant system information. It no subaction is given \texttt{port stats} prints the system information to \texttt{stdout}.

If the \texttt{submit} subaction is given then it will encode all the collected data as a \texttt{JSON} object. It then submits this via HTTP POST to a server specified in \texttt{macports.conf}.

\texttt{JSON} encoding is done though sub-procedures contained inside the procedure for the \texttt{port stats} action.

\subsection{Changes to port/port-help.tcl}

Added help entry for the \texttt{port stats} action describing proper usage.

\section{Data Format}

Transmitted data is encoded as a JSON object with four fields.

\begin{verbatim}
  {
      "id": "...",
      "os": {
          ...
      },
      "active_ports": [
          {...}, 
          ...
          {...}
      ],
      "inactive_ports": [
          {...}, 
          ...
          {...}
      ]
  }
\end{verbatim}

\begin{enumerate}
  \item id
  
  This is a string containing the user's UUID.
  
  \item os
  
  This is a JSON object containing information about the user's system.
  
  \begin{verbatim}
    "os": {
        "macports_version": "1.9.99",
        "osx_version": "10.6",
        "os_arch": "i386",
        "os_platform": "darwin",
        "build_arch": "x86_64",
        "gcc_version": "4.2.1",
        "xcode_version": "4.0"
    }
  \end{verbatim}
  
  \item active\_ports
  
  This is an array of json objects. Each object represents a single port.
  
  \begin{verbatim}
    "active_ports": [
        {
            "name": "aalib",
            "version": "1.4rc5_4"
        },
        {
            "variants": "nonls +",
            "name": "aspell",
            "version": "0.60.6_4"
        }
    ]
  \end{verbatim}
  
  \item inactive\_ports
  
  This is the same as active\_ports except that port objects represent installed inactive ports.
\end{enumerate}

\section{Server Side - Ruby on Rails}
\subsection{Database Schema}

\subsubsection {Categories table}

Imported from MPWA

\begin{verbatim}
  create_table "categories", :force => true do |t|
    t.string   "name"
    t.datetime "created_at"
    t.datetime "updated_at"
  end
\end{verbatim}

\textbf{Relationships and Validations}

\begin{verbatim}
  has_many :ports
  validates_presence_of :name
\end{verbatim}

\textbf{Changes from MPWA}
\begin{itemize}
  \item Validate presence of name
\end{itemize}

\subsection{Ports table}

Imported from MPWA

\begin{verbatim}
  create_table "ports", :force => true do |t|
    t.string   "name"
    t.string   "path"
    t.string   "version"
    t.text     "description"
    t.string   "licenses"
    t.integer  "category_id"
    t.text     "variants"
    t.string   "maintainers"
    t.string   "platforms"
    t.string   "categories"
    t.datetime "created_at"
    t.datetime "updated_at"
  end

  add_index "ports", ["name"], :name => "index_ports_on_name"
\end{verbatim}

\textbf{Relationships and Validations}
  
\begin{verbatim}
  has_one :category
  belongs_to :category
  has_many :installed_ports
  validates_presence_of :name, :version
\end{verbatim}

\textbf{Changes from MPWA}
\begin{itemize}
  \item Changed variant column to text type from string
  \item Added index on name column
  \item Validate presence of name and version
  \item has\_many installed ports
\end{itemize}

\subsubsection{installed\_ports table}

The installed\_ports table holds submitted port installation data. It keeps track of an installed port's version and variants as well as the id of the submitting user.

\begin{verbatim}
  create_table "installed_ports", :force => true do |t|
    t.integer  "port_id"
    t.string   "version"
    t.text     "variants"
    t.datetime "created_at"
    t.datetime "updated_at"
    t.integer  "user_id"
  end

  add_index "installed_ports", ["port_id"], :name => "index_installed_ports_on_port_id"
  add_index "installed_ports", ["user_id"], :name => "index_installed_ports_on_user_id"
\end{verbatim}

\textbf{Relationships and Validations}
  
\begin{verbatim}
  belongs_to :port
  has_one    :user
  
  validates_presence_of :user_id, :port_id, :version
\end{verbatim}

\subsubsection{os\_statistics table}

The os\_statistics table holds information about a user's system.

\begin{verbatim}
  create_table "os_statistics", :force => true do |t|
    t.datetime "created_at"
    t.datetime "updated_at"
    t.string   "macports_version"
    t.string   "osx_version"
    t.string   "os_arch"
    t.string   "os_platform"
    t.string   "build_arch"
    t.string   "xcode_version"
    t.string   "gcc_version"
    t.integer  "user_id"
  end

  add_index "os_statistics", ["user_id"], :name => "index_os_statistics_on_user_id"
\end{verbatim}

\begin{verbatim}
  belongs_to :port
  has_one    :user
  
  validates_presence_of :user_id, :port_id, :version
\end{verbatim}

\subsubsection{users table}

The users table holds UUIDs for each user.

\begin{verbatim}
  create_table "users", :force => true do |t|
    t.string   "uuid"
    t.datetime "created_at"
    t.datetime "updated_at"
  end
\end{verbatim}

\begin{verbatim}
  has_one  :os_statistic
  has_many :installed_ports
\end{verbatim}

\subsection{Submissions}

JSON encoded submissions are sent via HTTPS POST to the \texttt{/submissions} page. All data is stored in the \texttt{data} POST variable.

Submissions are stored on a month by month basis. Resubmissions in a given month cause that month's data to be updated.

Storing data happens as follows
\begin{enumerate}
  \item Attempt to find a user with the given UUID in the database. If no user is found then add a new entry
  \item Attempt to find an entry in the os\_statistics table for this user that was created this month. If no such entry is found then add an try for this month. If an entry is found then update it.
  \item For each submitted port verify that it is a valid port by checking to see if it exists in the \texttt{ports} table. If it does not exist then skip it. If it does exist then attempt to find an entry for the given user that was created this month. If an entry was found then update it, otherwise create a new entry.
\end{enumerate}
 
\subsection{OS Statistics Page}

The OS statistics page provides visualizations of the data in the \texttt{os\_statistics} table. It shows pie charts for each of
\begin{itemize}
  \item MacPorts Version
  \item OSX Versions
  \item OS Arch
  \item OS Platform
  \item Build Arch
  \item gcc Versions
  \item XCode Versions
\end{itemize}

These pie charts show the percentage of the user population running different versions (or arch / platform) in each category.

\subsection{Port Page}

Every port in the MacPorts repository has an associated port page. This page displays basic information about the port such as
\begin{itemize}
  \item Name
  \item Current version
  \item Licenses
  \item Categories
  \item Variants
\end{itemize}

This page also shows visualizations of the data in the \texttt{installed\_ports} page for this particular port.

It has the following
\begin{itemize}
  \item Line chart of installation counts over the past 12 months.
  \item Top versions over the past 12 months. This finds the top 5 most popular versions in use right now and tracks how their popularity has changed over the past 12 months. Popularity is measured by the number of installations of each version per month.
  \item Pie chart of all versions. This shows the distribution of all different versions in use right now. It will show you that \(x\%\) of users of this port are using version \(y\).
  \item Similarly to all versions, there is a pie chart of all variants in use.
\end{itemize}

\subsection{Installed Ports Page}

This page shows summary information for installed ports such as:
\begin{itemize}
  \item The number of participating users
  \item The number of ports in the MacPorts repository
  \item Average number of ports installed port user
  \item Most popular port this month and the number of installs
  \item Most popular port this year and the number of installs
  \item A bar chart of the top 25 most installed ports along with their install counts
  \item A table of the top 25 most installed ports along with their install counts
\end{itemize}

\subsection{Home Page}
The home has links to all other pages as well as a search area for ports. It also displays a line chart of the number of participating users over the past 12 months.

\end{document}

