reference, declarationdefinition
definition → references, declarations, derived classes, virtual overrides
reference to multiple definitions → definitions
unreferenced
    1
    2
    3
    4
    5
    6
    7
    8
    9
   10
   11
   12
   13
   14
   15
   16
   17
   18
   19
   20
   21
   22
   23
   24
   25
   26
   27
   28
   29
   30
   31
   32
   33
   34
   35
   36
   37
   38
   39
   40
   41
   42
   43
   44
   45
   46
   47
   48
   49
   50
   51
   52
   53
   54
   55
   56
   57
   58
   59
   60
   61
   62
   63
   64
   65
   66
   67
   68
   69
   70
   71
   72
   73
   74
   75
   76
   77
   78
   79
   80
   81
   82
   83
   84
   85
   86
   87
   88
   89
   90
   91
   92
   93
   94
   95
   96
   97
   98
   99
  100
  101
  102
  103
  104
  105
  106
  107
  108
  109
  110
  111
  112
  113
  114
  115
  116
  117
  118
  119
  120
  121
  122
  123
  124
  125
  126
  127
  128
  129
  130
  131
  132
  133
  134
  135
  136
  137
  138
  139
  140
  141
  142
  143
  144
  145
  146
  147
  148
  149
  150
  151
  152
  153
  154
  155
  156
  157
  158
  159
  160
  161
  162
  163
  164
  165
  166
  167
  168
  169
  170
  171
  172
  173
  174
  175
  176
  177
  178
  179
  180
  181
  182
  183
  184
  185
  186
  187
  188
  189
  190
  191
  192
  193
  194
  195
  196
  197
  198
  199
  200
  201
  202
  203
  204
  205
  206
  207
  208
  209
  210
=============================
How To Validate a New Release
=============================

.. contents::
   :local:
   :depth: 1

Introduction
============

This document contains information about testing the release candidates that will
ultimately be the next LLVM release. For more information on how to manage the
actual release, please refer to :doc:`HowToReleaseLLVM`.

Overview of the Release Process
-------------------------------

Once the release process starts, the Release Manager will ask for volunteers,
and it'll be the role of each volunteer to:

* Test and benchmark the previous release

* Test and benchmark each release candidate, comparing to the previous release and candidates

* Identify, reduce and report every regression found during tests and benchmarks

* Make sure the critical bugs get fixed and merged to the next release candidate

Not all bugs or regressions are show-stoppers and it's a bit of a grey area what
should be fixed before the next candidate and what can wait until the next release.

It'll depend on:

* The severity of the bug, how many people it affects and if it's a regression or a
  known bug. Known bugs are "unsupported features" and some bugs can be disabled if
  they have been implemented recently.

* The stage in the release. Less critical bugs should be considered to be fixed between
  RC1 and RC2, but not so much at the end of it.

* If it's a correctness or a performance regression. Performance regression tends to be
  taken more lightly than correctness.

.. _scripts:

Scripts
=======

The scripts are in the ``utils/release`` directory.

test-release.sh
---------------

This script will check-out, configure and compile LLVM+Clang (+ most add-ons, like ``compiler-rt``,
``libcxx``, ``libomp`` and ``clang-extra-tools``) in three stages, and will test the final stage.
It'll have installed the final binaries on the Phase3/Releasei(+Asserts) directory, and
that's the one you should use for the test-suite and other external tests.

To run the script on a specific release candidate run::

   ./test-release.sh \
        -release 3.3 \
        -rc 1 \
        -no-64bit \
        -test-asserts \
        -no-compare-files

Each system will require different options. For instance, x86_64 will obviously not need
``-no-64bit`` while 32-bit systems will, or the script will fail.

The important flags to get right are:

* On the pre-release, you should change ``-rc 1`` to ``-final``. On RC2, change it to ``-rc 2`` and so on.

* On non-release testing, you can use ``-final`` in conjunction with ``-no-checkout``, but you'll have to
  create the ``final`` directory by hand and link the correct source dir to ``final/llvm.src``.

* For release candidates, you need ``-test-asserts``, or it won't create a "Release+Asserts" directory,
  which is needed for release testing and benchmarking. This will take twice as long.

* On the final candidate you just need Release builds, and that's the binary directory you'll have to pack.

This script builds three phases of Clang+LLVM twice each (Release and Release+Asserts), so use
screen or nohup to avoid headaches, since it'll take a long time.

Use the ``--help`` option to see all the options and chose it according to your needs.


findRegressions-nightly.py
--------------------------

TODO

.. _test-suite:

Test Suite
==========

.. contents::
   :local:

Follow the `LNT Quick Start Guide <http://llvm.org/docs/lnt/quickstart.html>`__ link on how to set-up the test-suite

The binary location you'll have to use for testing is inside the ``rcN/Phase3/Release+Asserts/llvmCore-REL-RC.install``.
Link that directory to an easier location and run the test-suite.

An example on the run command line, assuming you created a link from the correct
install directory to ``~/devel/llvm/install``::

   ./sandbox/bin/python sandbox/bin/lnt runtest \
       nt \
       -j4 \
       --sandbox sandbox \
       --test-suite ~/devel/llvm/test/test-suite \
       --cc ~/devel/llvm/install/bin/clang \
       --cxx ~/devel/llvm/install/bin/clang++

It should have no new regressions, compared to the previous release or release candidate. You don't need to fix
all the bugs in the test-suite, since they're not necessarily meant to pass on all architectures all the time. This is
due to the nature of the result checking, which relies on direct comparison, and most of the time, the failures are
related to bad output checking, rather than bad code generation.

If the errors are in LLVM itself, please report every single regression found as blocker, and all the other bugs
as important, but not necessarily blocking the release to proceed. They can be set as "known failures" and to be
fix on a future date.

.. _pre-release-process:

Pre-Release Process
===================

.. contents::
   :local:

When the release process is announced on the mailing list, you should prepare
for the testing, by applying the same testing you'll do on the release candidates,
on the previous release.

You should:

* Download the previous release sources from http://llvm.org/releases/download.html.

* Run the test-release.sh script on ``final`` mode (change ``-rc 1`` to ``-final``).

* Once all three stages are done, it'll test the final stage.

* Using the ``Phase3/Release+Asserts/llvmCore-MAJ.MIN-final.install`` base, run the test-suite.

If the final phase's ``make check-all`` failed, it's a good idea to also test the
intermediate stages by going on the obj directory and running ``make check-all`` to find
if there's at least one stage that passes (helps when reducing the error for bug report
purposes).

.. _release-process:

Release Process
===============

.. contents::
   :local:

When the Release Manager sends you the release candidate, download all sources,
unzip on the same directory (there will be sym-links from the appropriate places
to them), and run the release test as above.

You should:

* Download the current candidate sources from where the release manager points you
  (ex. http://llvm.org/pre-releases/3.3/rc1/).

* Repeat the steps above with ``-rc 1``, ``-rc 2`` etc modes and run the test-suite
  the same way.

* Compare the results, report all errors on Bugzilla and publish the binary blob
  where the release manager can grab it.

Once the release manages announces that the latest candidate is the good one, you
have to pack the ``Release`` (no Asserts) install directory on ``Phase3`` and that
will be the official binary.

* Rename (or link) ``clang+llvm-REL-ARCH-ENV`` to the .install directory

* Tar that into the same name with ``.tar.gz`` extensioan from outside the directory

* Make it available for the release manager to download

.. _bug-reporting:

Bug Reporting Process
=====================

.. contents::
   :local:

If you found regressions or failures when comparing a release candidate with the
previous release, follow the rules below:

* Critical bugs on compilation should be fixed as soon as possible, possibly before
  releasing the binary blobs.

* Check-all tests should be fixed before the next release candidate, but can wait
  until the test-suite run is finished.

* Bugs in the test suite or unimportant check-all tests can be fixed in between
  release candidates.

* New features or recent big changes, when close to the release, should have done
  in a way that it's easy to disable. If they misbehave, prefer disabling them than
  releasing an unstable (but untested) binary package.