\documentclass[12pt]{article}
\usepackage{xcolor}
\begin{document}

\section*{JSS4832: Generalized Plackett-Luce Likelihoods by Robin
K. S. Hankin}


Below, the reviewer's comments are in black, and the replies to the
issues are in \textcolor{blue}{blue}.  I have indicated changes to
the manuscript where appropriate.

\textcolor{blue}{Short story: I have accommodated all the comments with
rewording and clarification.}

\subsection*{Reviewer's comments, 15 April 2023}


Dear Robin K.S. Hankin,

We have reached a decision regarding your submission to Journal of
Statistical Software, \verb+#4832+, ``Generalized Plackett-Luce
Likelihoods".

Unfortunately we have to tell you that we decided to reject in its current state.

However, we encourage you to resubmit the article after detailed
revision. Please check the journal management system for any
additional feedback:
https://www.jstatsoft.org/index.php/jss/authorDashboard/submission/4832. We
expect you to provide a point-by-point response to this feedback in
your revision.

Your sincerely,

The editorial team

The manuscript describes functionality of the hyper2 package for
Fitting and testing variants of the Placket-Luce ranking model.

1.1 Recommendation

The manuscript may be publishable in the journal after revisions have been
made.

1.2 General comments

G1 The introduction should provide an overview of the contents of the
manuscript. \\  \textcolor{blue}{New sentence added to manuscript:\\ \\
Here I consider such generalized Plackett-Luce likelihood functions,
and give an exact analysis of several simple cases.  I then show how
this class of likelihood functions may be applied to a range of
inference problems involving order statistics.  Illustrative examples,
drawn from Formula 1 motor racing, and track-and-field athletics, are
given.}\\ \\

In addition, the section headlines could be made more
informative. Currently, they are:

2. ``Some simple cases"
3. ``Mann-Whitney test"
4. ``Multiple competitors: javelin"
5. ``Formula 1 motor racing: the constructors' championship"



Maybe the headlines could provide more of a summary of what the
section contains. \\ \\ \textcolor{blue}{The headings are indeed somewhat
  uninformative and are now more in line with the remainder of the
  manuscript.  I have changed what was section 4 to a {\em sub}section
  with heading ``A generalization of the Mann-Whitney test using
  generalized Plackett-Luce likelihood'' which I think is in keeping
  with this comment.  The headings are now
  \begin{itemize}
  \item Exact analytical solution for some simple generalized
    Plackett-Luce likelihood functions
  \item An alternative to the Mann-Whitney test using generalized
    Plackett-Luce likelihood
  \item A generalization of the Mann-Whitney test using
    generalized Plackett-Luce likelihood (now a subsection)
  \item Formula 1 motor racing: the constructors' championship
\end{itemize}}
\rule{0mm}{10mm}

G2 There should be a comparison with other software.  Although the
author states in the introduction that with hyper2 ``It becomes
possible to analyse a wider range of likelihood functions than
standard Plackett-Luce", it is not clear to me which of the features
described later on are unique to the \verb+hyper2+ package and which
can also be obtained from other packages (like, e.g., PlackettLuce).

\textcolor{blue}{Although the hyper2 package has functionality not
  present in other implementations, the current manuscript focuses on
  the new generalization represented in the package by the hyper3
  class (that is, generalized Plackett-Luce likelihoods for order
  statistics with clones of identical Bradley-Terry strength).  Turner
  et. al. (2020), which I have cited, does a good job of setting out
  the functionality of hyper2 not present in other packages.  I think
  that an extended discussion of hyper2 unique functionality would be
  a distraction from the focus of the manuscript.  But I agree that a
  short statement documenting the unique aspects of the hyper2 package
  would be useful to JSS readers, and I have added a short paragraph
  to this effect.  Actually (in the spirit of full disclosure), I have
  added quite a lot of machinery to the package since Turner's 2020
  review and this gives me a welcome opportunity to mention some of
  it.}

G3 The manuscript should comment on the timing of the functions. For the
simple example on P. 4, the maxp(H) call takes more than 8 seconds on
my computer. This makes me wonder how timings would be for realistic
examples.

\textcolor{blue}{All optimization techniques become slow when used
  over high-dimensional spaces and {\tt maxp()} is no exception.  The
  examples chosen were well within its capabilities.  In addition, the
  default implementation of {\tt maxp()} in the package uses a *very*
  conservative set of assumptions, for example the default is to use
  $n=10$ slightly differing start points and perform (by default)
  $n-10$ independent optimizations, to check for convergence.  My
  thinking has matured over the last few months, and I have come to
  realise that this is absurdly over-engineered for the examples
  considered in the manuscript.  If one is sure that the convergence
  is guaranteed (as it is in the examples considered here), $n=1$ is
  perfectly adequate.  For example, using the case given on P4:
  {\tt\\ \\
> (system.time(mH <- maxp(H,n=1)))\\
   user  system elapsed \\
  0.355   0.003   0.357 \\
> maxp(H,n=1)\\
         a          b          c \\
0.21324090 0.08724824 0.69951086 \\
  }\\ \\
  The latest package version on github includes a discussion of this
  issue and I have added a brief discussion (``function {\tt maxp()}
  uses standard optimization techniques to locate the evaluate; it has
  access to first derivatives of the log-likelihood and as such has
  rapid convergence, if its objective function is reasonably
  smooth").}\\ \\


1.3 Specific comments\\ S1 I'm not claiming any expertise in English
language, but occasionally, I found the manuscript difficult to
understand. For example, in the abstract it says: ``The package
. . . furnishes idiom'' Couldn't that simply be: ``The package
. . . provides functionality''?

\textcolor{blue}{The original motivation for the package occurred when
  I had a large complicated set of Bradley-Terry observations which I
  wanted to translate into a likelihood function.  The R code became
  extremely long and unweildy, and was almost impossible to check.  It
  also became very slow to run.  What I wanted was a nice way to enter
  this kind of observations, in such a way as to maintain both
  computational efficiency [hence the use of the {\tt STL} map class]
  {\bf and} human readability.  I think that the phrase ``furnishes
  idiom" expresses this reasonably clearly.  But on reflection it does
  sound a bit odd and I have rephrased the abstract accordingly.}\\ \\


On P. 13, it says ``Here we use generalized Plackett-Luce to assess
the constructors' performance''. Isn't there a ``model'' missing?
(Plackett-Luce is just a name and cannot be used).
\textcolor{blue}{The original text was too terse and I have
  reworded.}\\ \\


S2 Figure 2: For readability, maybe use xlab = expression(p[a]) and maybe
instead of a legend, write AAB, ABA, BAA next to the line in the color
of that line.
\textcolor{blue}{Done, figure much improved}\\ \\


S3 P. 7: ``https:github.com/RobinHankin/hyper2'' should be
``https://github.com/RobinHankin/hyper2''.
\textcolor{blue}{Corrected}\\ \\


S4 P. 7: ``Wolf'' should be ``Wolfe''.
\textcolor{blue}{Corrected.}\\ \\


S5 P. 15, maxp(const2020, n=1): What does n do?
\textcolor{blue}{Argument $n$ discussed above under Timing}\\ \\

S6 P. 16, ``we see strong evidence for a real decrease in the
strength'': I'd tone that down.  The model is almost certainly wrong,
so it does not measure the ``real decrease.'' Also, power is unknown,
so it is difficult to say what ``evidence'' there is against H0. (See,
e.g., Gelman, Regression an other stories, Chapter 4: Time to unlearn
what you thought you knew about
statistics. https://statmodeling.stat.columbia.edu/2022/01/27/regression-
and-other-stories-free-pdf/)\\

\textcolor{blue}{I have not seen {\em Regression and other stories}
  before.  It is very good.  I have been studying chapter 4 as
  suggested.  I make some discursive responses here.\\ \\ I was
  fascinated by ``Researcher degrees of freedom''; I am familiar with
  the concept but have not seen this particular, very apposite, phrase
  used to describe it.  I would observe that my work is not in the
  same class as discussed in ``solving the Bible code puzzle'' [in
    which an unacceptably high amount of researcher DoF was
    documented].  The current manuscript is, for example, written to
  reproducibility standards and all data and indeed code held on
  public repositories [github and CRAN], open to scrutiny.  There is
  very little ``wiggle room'' in my approach: the data is unambiguous,
  and the model (Plackett-Luce) is clear and objective.\\ \\ I would
  observe that ROS uses the frequentist paradigm, a
  position adopted by a decreasing proportion of researchers.  For
  example, the notion of ``bias'' is discussed; but many workers
  (myself included) regard the bias of an estimator as unimportant: a
  biased estimator with a small mean square error is not without
  value.  I would observe that the concept of bias does not arise in
  either the Bayesian or likelihood paradigms as used in the current
  manuscript.\\ \\I am not sure what relevance ``power'' has here:
  this is the probability of {\em correctly} failing to reject the
  null when it is false.  In this case the null was not rejected, a
  type II error was not committed.  It is not clear whether a
  preferred alternative hypothesis exists (and further, there seems to
  be no natural ensemble to consider).\\ \\ The comment ``the model is
  almost certainly wrong'' is, I would argue, misleading.  All models
  are wrong.  But some are useful.  My view is that the truth of a
  statistical model [for some theological definition of truth] is a
  poor criterion: no random variable is truly Gaussian, for example;
  but the Student $t$-test is not useless as a result.  The issue is
  whether non-normality materially affects the properties of the test.
  Often this is not the case and the Gaussian approximation is
  adequate: the $t$-test is sufficiently robust to non-normality to be
  useful in a wide range of situations.  Is the $t$-test True?  No.
  Is it Useful?  Yes.  Consistent?  Yes.  Reliable?  Yes.  Easy to
  use?  Yes.  Familiar?  Yes.  Well-understood?  Yes.  Powerful?  Yes.
  The Student $t$-test has value, and so does Plackett-Luce.  My
  position is that the framework set out in the manuscript is
  plausible and intuitive.  I make no claim for absolute
  correctness.\\ \\ But in answer to your question, the small
  $p$-value and large likelihood ratio statistic certainly {\em is}
  evidence for a real decrease.  The hypothesis of no real strength
  change has low likelihood compared with the alternative of a drop in
  Bradley-Terry strength.  Either Plackett-Luce is seriously wrong, or
  PL is at least approximately correct, and some low-probability event
  has occurred which is misleading us.  Alternatively, as I suggest,
  Mercedes really is suffering disproportionately from some genuine
  race-affecting issue [porpoising? Lewis Hamilton getting older?
    unaccustomed F1 budget caps?].  The {\em strength} of this
  evidence is quantified by the small $p$-value (here less than 0.002)
  or alternatively a likelihood ratio of $e^{4.722}\simeq 112.4$,
  surely worthy of the epithet ``strong''.}\\ \\

S7 The references are lacking their DOIs.\\ \textcolor{blue}{Done
  (except for two books---for which I have followed APA6---and R
  itself, for which I used the result of {\tt citation()})} \\ \\

2 Review of software\\

The package includes many functions with a similar name. E.g., {\tt
  ordervec2supp()}, {\tt ordervec2supp3()}, {\tt ordervec2supp3a()},
{\tt ordertable2supp()} , {\tt ordertable2supp3()}. Maybe there could
be some overview of which to use for what
purpose.\\ \textcolor{blue}{An overview is given in {\tt
    ordervec2supp()} and {\tt ordervec2supp3()}, which are
  cross-referenced to each other.  Order tables are documented at {\tt
    ordertable.Rd}; order vectors at {\tt ordervec2supp.Rd}.  The
  functions mentioned by the referee all follow a consistent
  nomenclature: as it says in {\tt hyper3.Rd}, ``Functionality for
  {\tt hyper3} objects is generally indicated by adding a 3 to
  function names, eg {\tt gradient()} goes to {\tt gradient3()}''.  I
  have added a brief pointer in the manuscript to these package
  files.}


Section editor comments:

The author proposes an extension of the alreadty existing hyper2 package.
This new S3 class allows for some extended likelihood specifications.

\textcolor{blue}{I would agree with this.}\\ \\


Even though this is just a code snippet and the hyper2 package was
already published elsewhere, the author should provide some
introductory elaborations on the Plackett-Luce model. Along these
lines it would also be helpful to give a quick overview of the hyper2
package, and mention other related packages. As it is now it is
difficult for a reader who's not super familiar with PL-models and
hyper2, respectively, to follow the paper. Anyway, I find the runners
example as motivating example very illustrative, and actually like the
examples throughout the paper.\\ \textcolor{blue}{Thank you!  I too
  have enjoyed the runner paradigm.  But I am aware that the new
  likelihood function is novel and likely to be challenging for those
  not familiar with {\tt hyper2}.  I have added some discussion of
  {\tt hyper2} and its design principles.\\ \\I would not agree with
  the characterization as a code snippet.  The {\tt C++} code base for
  {\tt hyper3} functionality (318 lines) is similar to that for {\tt
    hyper2} (396 lines).  But admittedly {\tt hyper3} functionality
  does leverage many of the S3 methods used in {\tt hyper2}.}\\ \\

Some more aesthetic effort should be made in the code file (e.g.,
providing comments). The code looks like a simple purl()
extraction.\\ \textcolor{blue}{The JSS style file forbids comments
  within the code but encourages authors to comment in the normal
  LaTeX text.  Each code chunk is now either preceded by a description
  of its content, or followed with ``Above, we see\ldots'' or
  equivalent.}

\textcolor{blue}{Summary: the insightful and constructive comments of
  the review have led to a much improved submission.}

\end{document}

