1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
| Testing
=======
The LLDB test suite consists of three different kinds of test:
* Unit test. These are located under ``lldb/unittests`` and are written in C++
using googletest.
* Integration tests that test the debugger through the SB API. These are
located under ``lldb/packages/Python/lldbsuite`` and are written in Python
using ``dotest`` (LLDB's custom testing framework on top of unittest2).
* Integration tests that test the debugger through the command line. These are
locarted under `lldb/tests/Shell` and are written in a shell-style format
using FileCheck to verify its output.
All three test suites use the `LLVM Integrated Tester
<https://llvm.org/docs/CommandGuide/lit.html>`_ (lit) as their test driver. The
test suites can be run as a whole or separately.
Many of the tests are accompanied by a C (C++, ObjC, etc.) source file. Each
test first compiles the source file and then uses LLDB to debug the resulting
executable.
.. contents::
:local:
.. note::
On Windows any invocations of python should be replaced with python_d, the
debug interpreter, when running the test suite against a debug version of
LLDB.
.. note::
On NetBSD you must export ``LD_LIBRARY_PATH=$PWD/lib`` in your environment.
This is due to lack of the ``$ORIGIN`` linker feature.
Running the Full Test Suite
---------------------------
The easiest way to run the LLDB test suite is to use the ``check-lldb`` build
target.
By default, the ``check-lldb`` target builds the test programs with the same
compiler that was used to build LLDB. To build the tests with a different
compiler, you can set the ``LLDB_TEST_COMPILER`` CMake variable.
It is possible to customize the architecture of the test binaries and compiler
used by appending ``-A`` and ``-C`` options respectively to the CMake variable
``LLDB_TEST_USER_ARGS``. For example, to test LLDB against 32-bit binaries
built with a custom version of clang, do:
::
> cmake -DLLDB_TEST_USER_ARGS="-A i386 -C /path/to/custom/clang" -G Ninja
> ninja check-lldb
Note that multiple ``-A`` and ``-C`` flags can be specified to
``LLDB_TEST_USER_ARGS``.
Running a Single Test Suite
---------------------------
Each test suite can be run separately, similar to running the whole test suite
with ``check-lldb``.
* Use ``check-lldb-unit`` to run just the unit tests.
* Use ``check-lldb-api`` to run just the SB API tests.
* Use ``check-lldb-shell`` to run just the shell tests.
You can run specific subdirectories by appending the directory name to the
target. For example, to run all the tests in ``ObjectFile``, you can use the
target ``check-lldb-shell-objectfile``. However, because the unit tests and API
tests don't actually live under ``lldb/test``, this convenience is only
available for the shell tests.
Running a Single Test
---------------------
The recommended way to run a single test is by invoking the lit driver with a
filter. This ensures that the test is run with the same configuration as when
run as part of a test suite.
::
> ./bin/llvm-lit -sv lldb/test --filter <test>
Because lit automatically scans a directory for tests, it's also possible to
pass a subdirectory to run a specific subset of the tests.
::
> ./bin/llvm-lit -sv tools/lldb/test/Shell/Commands/CommandScriptImmediateOutput
For the SB API tests it is possible to forward arguments to ``dotest.py`` by
passing ``--param`` to lit and setting a value for ``dotest-args``.
::
> ./bin/llvm-lit -sv tools/lldb/test --param dotest-args='-C gcc'
Below is an overview of running individual test in the unit and API test suites
without going through the lit driver.
Running a Specific Test or Set of Tests: API Tests
--------------------------------------------------
In addition to running all the LLDB test suites with the ``check-lldb`` CMake
target above, it is possible to run individual LLDB tests. If you have a CMake
build you can use the ``lldb-dotest`` binary, which is a wrapper around
``dotest.py`` that passes all the arguments configured by CMake.
Alternatively, you can use ``dotest.py`` directly, if you want to run a test
one-off with a different configuration.
For example, to run the test cases defined in TestInferiorCrashing.py, run:
::
> ./bin/lldb-dotest -p TestInferiorCrashing.py
::
> cd $lldb/test
> python dotest.py --executable <path-to-lldb> -p TestInferiorCrashing.py ../packages/Python/lldbsuite/test
If the test is not specified by name (e.g. if you leave the ``-p`` argument
off), all tests in that directory will be executed:
::
> ./bin/lldb-dotest functionalities/data-formatter
::
> python dotest.py --executable <path-to-lldb> functionalities/data-formatter
Many more options that are available. To see a list of all of them, run:
::
> python dotest.py -h
Running a Specific Test or Set of Tests: Unit Tests
---------------------------------------------------
The unit tests are simple executables, located in the build directory under ``tools/lldb/unittests``.
To run them, just run the test binary, for example, to run all the Host tests:
::
> ./tools/lldb/unittests/Host/HostTests
To run a specific test, pass a filter, for example:
::
> ./tools/lldb/unittests/Host/HostTests --gtest_filter=SocketTest.DomainListenConnectAccept
Running the Test Suite Remotely
-------------------------------
Running the test-suite remotely is similar to the process of running a local
test suite, but there are two things to have in mind:
1. You must have the lldb-server running on the remote system, ready to accept
multiple connections. For more information on how to setup remote debugging
see the Remote debugging page.
2. You must tell the test-suite how to connect to the remote system. This is
achieved using the ``--platform-name``, ``--platform-url`` and
``--platform-working-dir`` parameters to ``dotest.py``. These parameters
correspond to the platform select and platform connect LLDB commands. You
will usually also need to specify the compiler and architecture for the
remote system.
Currently, running the remote test suite is supported only with ``dotest.py`` (or
dosep.py with a single thread), but we expect this issue to be addressed in the
near future.
Debugging Test Failures
-----------------------
On non-Windows platforms, you can use the ``-d`` option to ``dotest.py`` which
will cause the script to wait for a while until a debugger is attached.
Debugging Test Failures on Windows
----------------------------------
On Windows, it is strongly recommended to use Python Tools for Visual Studio
for debugging test failures. It can seamlessly step between native and managed
code, which is very helpful when you need to step through the test itself, and
then into the LLDB code that backs the operations the test is performing.
A quick guide to getting started with PTVS is as follows:
#. Install PTVS
#. Create a Visual Studio Project for the Python code.
#. Go to File -> New -> Project -> Python -> From Existing Python Code.
#. Choose llvm/tools/lldb as the directory containing the Python code.
#. When asked where to save the .pyproj file, choose the folder ``llvm/tools/lldb/pyproj``. This is a special folder that is ignored by the ``.gitignore`` file, since it is not checked in.
#. Set test/dotest.py as the startup file
#. Make sure there is a Python Environment installed for your distribution. For example, if you installed Python to ``C:\Python35``, PTVS needs to know that this is the interpreter you want to use for running the test suite.
#. Go to Tools -> Options -> Python Tools -> Environment Options
#. Click Add Environment, and enter Python 3.5 Debug for the name. Fill out the values correctly.
#. Configure the project to use this debug interpreter.
#. Right click the Project node in Solution Explorer.
#. In the General tab, Make sure Python 3.5 Debug is the selected Interpreter.
#. In Debug/Search Paths, enter the path to your ninja/lib/site-packages directory.
#. In Debug/Environment Variables, enter ``VCINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\``.
#. If you want to enabled mixed mode debugging, check Enable native code debugging (this slows down debugging, so enable it only on an as-needed basis.)
#. Set the command line for the test suite to run.
#. Right click the project in solution explorer and choose the Debug tab.
#. Enter the arguments to dotest.py.
#. Example command options:
::
--arch=i686
# Path to debug lldb.exe
--executable D:/src/llvmbuild/ninja/bin/lldb.exe
# Directory to store log files
-s D:/src/llvmbuild/ninja/lldb-test-traces
-u CXXFLAGS -u CFLAGS
# If a test crashes, show JIT debugging dialog.
--enable-crash-dialog
# Path to release clang.exe
-C d:\src\llvmbuild\ninja_release\bin\clang.exe
# Path to the particular test you want to debug.
-p TestPaths.py
# Root of test tree
D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test
::
--arch=i686 --executable D:/src/llvmbuild/ninja/bin/lldb.exe -s D:/src/llvmbuild/ninja/lldb-test-traces -u CXXFLAGS -u CFLAGS --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe -p TestPaths.py D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test --no-multiprocess
|