File feature-upstream-testsuite-enhancement-2.patch of Package erlang.30223

Overview Repositories Revisions Requests Users Attributes Meta

File feature-upstream-testsuite-enhancement-2.patch of Package erlang.30223

From 381c19a6bc853f63a73e32d137a7787a0a24c3c2 Mon Sep 17 00:00:00 2001
From: Lukas Larsson <lukas@erlang.org>
Date: Tue, 21 Dec 2021 11:09:38 +0100
Subject: [PATCH] otp: Add DEVELOPMENT howto and complete `make test` support

This commit polishes a lot of the make system to be more
robust and hopefully user friendly. A new DEVELOPMENT howto
guide has been added with instructions on how to work with
the Erlang/OTP code and the `make test` functionality has
been fixed so that it should work for all applications.

In order to solve the problem of some applications relying
on being tested using a release, each application has been
instrumented in its Makefile so that it knows if a release
has to be done or not.
---
 .gitignore                         |   2 +
 CONTRIBUTING.md                    |  37 ++-
 HOWTO/DEVELOPMENT.md               | 423 +++++++++++++++++++++++++++++
 HOWTO/INSTALL.md                   |  57 ++--
 HOWTO/TESTING.md                   | 186 +++++++------
 Makefile.in                        |  19 +-
 README.md                          |  46 ++--
 erts/Makefile                      |  10 +-
 lib/common_test/test_server/ts.erl |   5 +-
 make/app_targets.mk                |   3 +-
 make/test_target_script.sh         | 149 +++++++---
 11 files changed, 738 insertions(+), 199 deletions(-)
 create mode 100644 HOWTO/DEVELOPMENT.md

Index: otp-OTP-22.2.7/CONTRIBUTING.md
===================================================================
--- otp-OTP-22.2.7.orig/CONTRIBUTING.md
+++ otp-OTP-22.2.7/CONTRIBUTING.md
@@ -47,7 +47,7 @@ Make sure you create a new branch for yo
 Give the branch a short but descriptive name, like `stdlib/lists-length-fix`.
 Never do your work directly on `maint` or `master`.
 
-## Fixing a bug
+### Fixing a bug
 
 * In most cases, pull requests for bug fixes should be based on the `maint` branch.
 There are exceptions, for example corrections to bugs that have been introduced in the `master` branch.
@@ -59,12 +59,12 @@ There are exceptions, for example correc
 * For applications without a test suite in the git repository, it would be appreciated if you provide a
 small code sample in the commit message or email a module that will provoke the failure.
 
-## Adding a new feature
+### Adding a new feature
 
 * In most cases, pull requests for new features should be based on the `master` branch.
 
 * It is recommended to discuss new features on
-[the erlang-questions mailing list](http://erlang.org/mailman/listinfo/erlang-questions),
+* It is recommended to discuss new features in the [erlang forums](https://erlangforums.com),
 especially for major new features or any new features in ERTS, Kernel, or STDLIB.
 
 * It is important to write a good commit message explaining **why** the feature is needed.
@@ -90,11 +90,10 @@ feature one or two releases beforehand.
 can be included in OTP. Major changes or new features in ERTS, Kernel, or STDLIB will need an EEP or at least
 a discussion on the mailing list.
 
-## Before you submit your pull request
+### Before you submit your pull request
 
 * Make sure existing test cases don't fail. It is not necessary to run all tests (that would take many hours),
 but you should at least run the tests for the application you have changed.
-See [Running tests](https://github.com/erlang/otp/wiki/Running-tests).
 
 Make sure that your branch contains clean commits:
 
@@ -120,7 +119,11 @@ Check your coding style:
 
 * In most code (Erlang and C), indentation is 4 steps. Indentation using only spaces is **strongly recommended**.
 
-### Configuring Emacs
+See the [Testing](https://github.com/erlang/otp/blob/master/HOWTO/TESTING.md) and
+[Development](https://github.com/erlang/otp/blob/master/HOWTO/DEVELOPMENT.md) howtos
+for details on how to use run tests and use the Erlang/OTP make system.
+
+#### Configuring Emacs
 
 If you use Emacs, use the Erlang mode, and add the following lines to `.emacs`:
 
Index: otp-OTP-22.2.7/HOWTO/DEVELOPMENT.md
===================================================================
--- /dev/null
+++ otp-OTP-22.2.7/HOWTO/DEVELOPMENT.md
@@ -0,0 +1,423 @@
+# Developing Erlang/OTP
+
+The Erlang/OTP development repository is quite large and the make system
+contains a lot of functionality to help when a developing. This howto
+will try to showcase the most important features of the make system.
+
+The guide is mostly aimed towards development on a Unix platform, but
+most things should work also work using WSL on Windows. The guide also
+assumes that you are working in the git repositiory. Many of the
+scripts and tools described here are not available in the prebuilt tar
+archive that you can download for each release.
+
+*WARNING*: Only some of APIs mentioned in this guide are supported. This
+means that they may be removed or changed without prior notice, so do
+not depend on them in CI. For supported make targets see the
+[Install howto](INSTALL.md) and the [Testing howto](TESTING.md).
+
+The make system is not always as robust as one might like, so if for
+any reason something does not work, try doing a `git clean -Xfdq` and
+start from the beginning again. This normally only needs to be done when
+you jump in between different git branches, but it can a good thing to
+keep in mind whenever things do not work as you expect them to.
+
+*NOTE*: This instructions may vary for different versions of Erlang/OTP,
+so make sure to read the instructions for the version that you are working
+with.
+
+1. [Short version](#short-version)
+2. [Preparations](#preparations)
+    1. [Faster builds](#faster-builds)
+3. [Configuring](#configuring)
+    1. [Help](#help)
+4. [Building and testing](#building-and-testing)
+    1. [Build and test a specific application](#build-and-test-a-specific-application)
+    2. [Preloaded and Primary Bootstrap](#preloaded-and-primary-bootstrap)
+    3. [Types and Flavors](#types-and-Flavors)
+    4. [cerl](#cerl)
+    5. [Static analysis](#static-analysis)
+5. [Running test cases](#running-test-cases)
+6. [Writing and building documentation](#writing-and-building-documentation)
+
+## Short version
+
+First make sure you have done all [preparations](#preparations) then
+do this:
+
+```bash
+git clone -b maint git@github.com:erlang/otp
+cd otp && export ERL_TOP=`pwd`
+./otp_build configure && make
+```
+
+When you have done changes, added tests or updated documentation, build and test like this:
+
+```bash
+cd lib/$APPLICATION_NAME
+make            # Rebuid application
+make test       # Run application tests
+make dialyzer   # Run dialyzer
+make docs       # Build the docs
+make xmllint    # Run xmllint on the docs
+```
+
+## Preparations
+
+Before you start working you need to clone the Erlang/OTP git repository
+and install any dependencies that you do not have. See
+[Required Utilities](INSTALL.md#required-utilities) and
+[Optional Utilities](INSTALL.md#optional-utilities) in [INSTALL.md](INSTALL.md)
+for a list of utilities to install. (Windows has its own [INSTALL Guide](INSTALL-WIN32.md)
+with its own [Required Utilities](INSTALL-WIN32.md#tools-you-need-and-their-environment)).
+
+Then you need to set `ERL_TOP` to point at the repository you are developing in.
+Not all make commands needs this environment variable set, but many do so it is
+good to get into the habit of always setting it.
+
+```bash
+cd /path/to/repository/otp
+export ERL_TOP=`pwd`
+```
+
+Make sure that you have read the [Contributing to Erlang/OTP](../CONTRIBUTING.md)
+guide if you intend to make a contribution to Erlang/OTP.
+
+### Faster builds
+
+Both `configure` and `make` take advantage of running in parallel if told to,
+so in order to speed up your development environment make sure to set:
+
+```bash
+## Change N to be at least the number of cores or hyper-threads available
+export MAKEFLAGS=-jN
+```
+
+The Erlang compiler can be run using a [Compile Server](https://www.erlang.org/doc/man/erlc.html#compile-server),
+this can cut from the total build time of Erlang/OTP by quite a lot,
+especially if you have a relatively slow machine.
+To enable set this environment variable:
+
+```bash
+export ERLC_USE_SERVER=true
+```
+
+Re-building all application in Erlang/OTP can take a while so it is possible
+to build only a subset of the applications. This is done by setting either
+`OTP_SMALL_BUILD` or `OTP_TINY_BUILD` to `true` when doing make at the top
+level. However, the simplest way is probably to just use the `./otp_build`
+wrapper that takes the options `-t` (tiny) or `-a` (all) and defaults to
+a small build.
+
+```bash
+# You need to have done ./configure before calling make or boot.
+OTP_TINY_BUILD=true make      ## Equivalent to ./otp_build boot -t
+OTP_SMALL_BUILD=true make     ## Equivalent to ./otp_build boot
+./otp_build boot -a           ## Equivalent to make
+```
+
+## Configuring
+
+You run configure by issuing the command:
+
+```bash
+./otp_build configure
+```
+
+On all operating systems except Windows you can also just run:
+
+```bash
+./configure
+```
+
+If you change any `Makefile`s you will need to re-run configure.
+If you update any `configure.ac` scripts you need to
+[update the configure scripts](INSTALL.md#updating-configure-scripts).
+
+### Help
+
+The toplevel configure will give help about the features that it provides.
+To get a full list of all features you need to use:
+
+```bash
+./configure --help=r
+```
+
+There is documentation for what most of the options mean in the
+[INSTALL.md](INSTALL.md#Configuring) howto.
+
+## Building and testing
+
+After you have done configure, you can do
+
+```bash
+make
+```
+
+on the top of this repository. That will compile all of Erlang/OTP.
+
+You can also build a specific application:
+
+```bash
+make stdlib
+make common_test
+```
+
+These make commands do not manage any dependencies, so if an application needs
+something from another you need to make sure that it is built. It is therefore
+good practice to first build all of Erlang/OTP and then build just the one that
+you are updating.
+
+You can also run tests from the top:
+
+```bash
+make test                # Run all tests, takes a **very** long time
+make stdlib_test         # Run only stdlib tests, takes less time
+                         # Run only lists_SUITE, takes even less time
+make stdlib_test ARGS="-suite lists_SUITE"
+                         # Run only member testcase in lists_SUITE
+make stdlib_test ARGS="-suite lists_SUITE -case member"
+```
+
+See [ct_run](https://www.erlang.org/doc/man/ct_run.html#) for a list of all options
+that you can pass to ARGS.
+
+You can run static analysis test:
+
+```bash
+make dialyzer            # Checks all of Erlang/OTP source code
+make xmllint             # Checks all documentation for xmllint errors
+```
+
+Most of the above targets also works for a "phony" target called `emulator` that
+represents erts and all its tools. So you can do this:
+
+```bash
+make emulator            # Build erts, epmd etc
+make emulator_test       # Run all emulator tests
+```
+
+If you want to pass a run-time flag to the emulator running the tests you can
+use the `ERL_ARGS` flags to `make test`. For example if you want to run tests
+using [off heap message queue data](https://www.erlang.org/doc/man/erlang.html#process_flag_message_queue_data)
+for all process you would do this:
+
+```bash
+ERL_ARGS="+hmqd off_heap" make emulator_test
+```
+
+### Build and test a specific application
+
+You can also build the application from within itself. Like this:
+
+```bash
+cd lib/stdlib && make
+```
+
+Each application has a bunch of make targets that you can use.
+
+```bash
+make                                # build all source for this application
+make test                           # run all tests for this application
+make test ARGS="-suite lists_SUITE" # run the lists_SUITE tests
+make dialyzer                       # run dialyzer for this application
+make docs                           # build all docs for this application
+make docs DOC_TARGETS="html"        # build html docs for this application
+make xmllint                        # run xmllint on the docs for this application
+```
+
+If you want to view what the documentation looks like for only your application
+you can do this:
+
+```bash
+(cd doc/src && make local_docs)
+```
+
+and then view `doc/html/index.html`.
+
+### Preloaded and Primary Bootstrap
+
+The Erlang code loader and compiler are written in Erlang, so in order to
+[bootstrap](https://en.wikipedia.org/wiki/Bootstrapping_(compilers))
+the system a number of compiled `.beam` files are commited into the
+Erlang/OTP git repository.
+
+The Erlang code located in [erts/preloaded/src](../erts/preloaded/src)
+is compiled into the VM and used to load enough code so that the code
+loader in the `kernel` application can be loaded. If you update any of
+that code you need to do a special preloaded update for the changes to
+take effect. This is done like this:
+
+```bash
+./otp_build update_preloaded [--no-commit]
+make  # Need to rebuild system after the preloaded has been updated
+```
+
+You need to have a working Erlang compiler in your path for this to work.
+In order to be able to compile the Erlang/OTP source code, there also needs
+to be a basic Erlang compiler committed into git. This is what is called the
+primary bootstrap. It is quite rare that you need to update this, but if you
+are extending the Erlang language and would like to use the new extensions
+in the Erlang/OTP source code you it needs to be updated. As an example, when
+we added `maps` to Erlang we first needed to have a commited primary bootstrap
+that could compile code with maps, before we actually could use maps anywhere.
+To update the primary bootstrap you do like this:
+
+```bash
+./otp_build update_primary [--no-commit]
+```
+
+*NOTE*: When submitting a PR to Erlang/OTP you will be asked to not include
+any commit updating preloaded or the primary bootstrap. This is because we
+cannot review the contents of binary files and thus cannot make sure they do
+not contain any malicious data.
+
+### Types and Flavors
+
+Erlang can be built using different types and flavors. Mostly the types and
+flavors change how the Erlang VM itself is built, but some also effect how
+application are built. Some of the types/flavors are:
+
+* Types
+  * opt (default)
+  * debug
+  * lcnt
+  * valgrind
+  * asan
+  * gcov
+* Flavor
+  * emu
+  * jit (default if available)
+
+To build using a type and or flavor you just pass it as a variable to make.
+For example:
+
+```bash
+make TYPE=debug
+make FLAVOR=emu
+make TYPE=lcnt FLAVOR=emu
+```
+
+As you can see it is possible to combine type and flavor to create many different
+versions of Erlang. You can then run these different versions by passing the
+`-emu_type` and/or `-emu_flavor` flags to `erl`. That is:
+
+```bash
+erl -emu_type lcnt
+erl -emu_flavor emu -emu_type debug
+```
+
+When running valgrind, asan or gcov those tools create special output files that
+need to be processed. To work with these files there is a special `erl` program
+called `cerl` that is only available in the source tree. You can read more about
+it in the [cerl section](#cerl) later in this guide.
+
+If you want to run the tests with a special flavor or type, the easiest way to
+do that is by setting the TYPE or FLAVOR when calling make. For example if you
+want to run the emulator tests using the debug emulator you can do it like this:
+
+```bash
+make emulator_test TYPE=debug
+```
+
+*NOTE*: Before you run tests using a TYPE or FLAVOR you need to build the **entire**
+Erlang/OTP repo using that TYPE or FLAVOR. That is `make TYPE=debug` for the example
+above.
+
+### cerl
+
+`cerl` is a program available in `$ERL_TOP/bin/` that has a number of features
+useful when developing the Erlang run-time system. It work just as normal `erl`,
+but accepts a couple of extra command line switches. Any other command line arguments
+passed to `cerl` will be passed on the Erlang as normal. The extra command line
+switches are:
+
+* -debug
+  * Start a debug run-time system.
+* -lcnt
+  * Start a lock count run-time system.
+* -valgrind
+  * Start valgrind with the correct settings and use the `valgrind` [type](types-and-flavors).
+  * Set environment variable `VALGRIND_LOG_XML` to true if want xml valgrind logs.
+  * Set environment variable `VALGRIND_LOG_DIR` to where you want valgrind logs.
+  * Set environment variable `VALGRIND_MISC_FLAGS` for any extra valgrind flags you want to pass.
+* -asan
+  * Start [Clang Address Sanitizer](https://clang.llvm.org/docs/AddressSanitizer.html)
+    with the the correct settings and use the `asan` [type](types-and-flavors).
+  * Set environment variable `ASAN_LOG_DIR` to where you want the logs.
+  * Set environment variable `ASAN_OPTIONS` for any extra asan options you want to pass.
+* -gcov
+  * Start a gcov run-time system.
+* -gdb
+  * Start an Emacs gdb debugging session. Can be combined with -debug.
+* -core /path/to/core/file
+  * Start an Emacs gdb debugging session for the core specified.
+* -rgdb
+  * Start a gdb debugging session in the current terminal. Can be combined with -debug.
+* -rcore /path/to/core/file
+  * Start a gdb debugging session in the current terminal for the core specified.
+* -lldb
+  * Start a lldb debugging session in the current terminal.
+* -rr
+  * Start Erlang under [rr](https://rr-project.org/) to record all events. Can be combined with -debug.
+* -rr replay [session]
+  * Load a recording session using `rr replay`, if no session is specified the latest run session is laoded.
+
+If you want to run tests using `cerl` (for example if you want to run asan on
+the nif_SUITE in emulator) you cannot use the `make test` approach to testing
+as that uses `ct_run` under the hood and `ct_run` does not support customizing
+the emulator start script. Instead you need to use the approach described in
+[Run tests with Address Sanitizer](INSTALL.md#run-tests-with-address-sanitizer).
+
+
+### Static analysis
+
+From the top level of Erlang/OTP you can run:
+
+```bash
+make xmllint
+make dialyzer
+make format-check
+```
+
+This will check that the documentation is correct and that there are no
+dialyzer errors.
+
+## Running test cases
+
+There is a detailed description about how to run tests in [TESTING.md](TESTING.md).
+
+## Writing and building documentation
+
+Most of the Erlang/OTP documentation is written in XML files located in
+`lib/$APPLICATION_NAME/doc/src`. The format of the XML is described in the
+[ErlDocgen User's Guide](https://www.erlang.org/doc/apps/erl_docgen/users_guide.html).
+
+There is also some documentation that is written using [edoc](https://www.erlang.org/doc/man/edoc.html).
+
+To view the documentation the simplest way is to release it. *NOTE*: The Erlang/OTP
+repository needs to have been [built](#building-and-testing) before you can build
+the documentation.
+
+```bash
+make release_docs
+```
+
+and then you can view `release/*/doc/index.html` in your favourite browser and
+make sure that it looks nice.
+
+This takes a while though and to speed up the edit-view cycle you can either
+limit what parts of the documentation is built using `DOC_TARGETS`. For example:
+
+```bash
+make release_docs DOC_TARGETS=html
+```
+
+The different `DOC_TARGETS` built are `html`, `man`, `pdf` and `chunks`.
+
+You can also build the docs only for a single application. For example:
+
+```bash
+cd lib/stdlib/doc/src && make local_docs DOC_TARGETS=html
+```
+
+and then view the results at `lib/stdlib/doc/html/index.html`.
Index: otp-OTP-22.2.7/HOWTO/INSTALL.md
===================================================================
--- otp-OTP-22.2.7.orig/HOWTO/INSTALL.md
+++ otp-OTP-22.2.7/HOWTO/INSTALL.md
@@ -50,7 +50,6 @@ These are the tools you need in order to
 
 *   An `install` program that can take multiple file names.
 
-
 Optional Utilities
 ------------------
 
@@ -88,8 +87,6 @@ also find the utilities needed for build
 
     Further instructions on wxWidgets, read [Building with wxErlang][].
 
-
-
 ### Building Documentation ###
 
 *   `xsltproc` -- A command line XSLT processor.
@@ -101,8 +98,6 @@ also find the utilities needed for build
 *   `fop` -- Apache FOP print formatter (requires Java). Can be downloaded
     from <http://xmlgraphics.apache.org/fop>.
 
-
-
 How to Build and Install Erlang/OTP
 -----------------------------------
 
@@ -145,14 +140,12 @@ set. If you get errors when building, tr
 
     $ export LANG=C   # Assuming bash/sh
 
-
 ### Building ###
 
 Build the Erlang/OTP release.
 
     $ make
 
-
 ### Testing ###
 
 Before installation you should test whether your build is working properly
@@ -181,13 +174,11 @@ The following command will install the r
 
     $ make install
 
-
 ### Running ###
 
 You should now have a working release of Erlang/OTP!
 Jump to [System Principles][] for instructions on running Erlang/OTP.
 
-
 ### How to Build the Documentation ###
 
 Make sure you're in the top directory in the source tree.
@@ -789,7 +780,7 @@ Use `hipe:help_options/0` to print out t
    [html documentation]: http://www.erlang.org/download/otp_doc_html_%OTP-VSN%.tar.gz
    [man pages]: http://www.erlang.org/download/otp_doc_man_%OTP-VSN%.tar.gz
    [the released source tar ball]: http://www.erlang.org/download/otp_src_%OTP-VSN%.tar.gz
-   [System Principles]: ../system_principles/system_principles
+   [System Principles]: system/system_principles:system_principles
    [native build]: #How-to-Build-and-Install-ErlangOTP
    [cross build]: INSTALL-CROSS.md
    [Required Utilities]: #Required-Utilities
Index: otp-OTP-22.2.7/HOWTO/TESTING.md
===================================================================
--- otp-OTP-22.2.7.orig/HOWTO/TESTING.md
+++ otp-OTP-22.2.7/HOWTO/TESTING.md
@@ -5,17 +5,24 @@ Before you start testing you need to hav
 are going to test in your path. See [$ERL_TOP/HOWTO/INSTALL.md][] for
 instructions on how to build an Erlang release.
 
+*NOTE*: This instructions may vary for different versions of Erlang/OTP,
+so make sure to read the instructions for the version that you are testing.
+
 Short version
 -------------
-Move to the top directory of the Erlang release you want to test, i.e.
-cd /ldisk/work/otp
+Move to the top directory of the Erlang release you want to test, for example:
+cd $HOME/git/otp
 
-    export ERL_TOP=`pwd`
-	./otp_build setup -a
-	export PATH=`pwd`/bin:$PATH
-	./otp_build tests
-	cd release/tests/test_server
-	erl -s ts install -s ts run all_tests -s init stop
+```bash
+export ERL_TOP=`pwd`            # Define where the build root is
+./configure && make             # Build all of Erlang/OTP
+make test                       # Test all of Erlang/OTP
+```
+
+The tests will take a long time to finish and will print a lot of logs to the
+console even if tests pass. A full run takes close about 6 hours on a relatively
+modern machine. See [Running tests while developing][] for details on how to run
+only a subset of the tests.
 
 Where are the tests
 -------------------
@@ -46,14 +53,57 @@ the erl_interface tests use this feature
 [configuring the tests][]. These `Makefile`s are later run by the test suite
 to compile whatever platform specific code the tests need to run.
 
+Running tests while developing
+------------------------------
+
+The `make test` command works when the current directory contains a directory
+called test and in the root directory of the source code tree.
+
+Below are some examples that illustrate how `make test` can be
+used:
+
+```bash
+# ERL_TOP needs to be set correctly
+cd /path/to/otp
+export ERL_TOP=`pwd`
+
+# Build Erlang/OTP
+#
+# Note that make test will only compile test code except when
+# make test is executed from $ERL_TOP.
+./otp_build setup -a
+
+# Run all test suites for an application
+(cd $ERL_TOP/lib/asn1 && make test)
+make asn1_test
+
+# Run a test suite (The ARGS variable is passed to ct_run)
+(cd $ERL_TOP/lib/stdlib && make test ARGS="-suite ets_SUITE")
+make stdlib_test ARGS="-suite ets_SUITE"
+
+# Run a test case
+(cd $ERL_TOP/erts/emulator && make test ARGS="-suite binary_SUITE -case deep_bitstr_lists")
+make emulator_test ARGS="-suite binary_SUITE -case deep_bitstr_lists"
+
+# Run all tests
+#
+# When executed from $ERL_TOP, "make test" will first release and
+# configure all tests and then attempt to run all tests with `ts:run`.
+# This will take several hours.
+(cd $ERL_TOP && make test)
+```
+
+For more examples see [DEVELOPMENT.md](DEVELOPMENT.md)
+
 Releasing tests
 ---------------
 
-If you cannot use [ct_run][] in the source tree you have to release the tests
-into a common test directory. The easiest way to do this is to use `otp_build`
-like this:
+When not testing in the source tree, you first need to release all tests.
+The easiest way to do this is to use `otp_build` like this:
 
-    export ERL_TOP=`pwd`; ./otp_build tests
+```bash
+export ERL_TOP=`pwd`; ./otp_build tests
+```
 
 This will release all tests in Erlang/OTP to `$ERL_TOP/release/tests/`. If you
 want to change the directory where the tests are released to use the `TESTROOT`
@@ -99,91 +149,51 @@ All variables created by `ts:install()`
 
 To run all test suites go to `$TESTROOT/test_server` fire up an Erlang shell and type:
 
-    ts:run().
+```erlang
+ts:run().
+```
 
 Note that running all tests will require several hours, so you may want to run
 the test cases for a single application
 
-    ts:run(Application, [batch]).
+```erlang
+ts:run(Application, [batch]).
+```
 
 or even part of the test suite for an application, for example
 
-    ts:run(emulator, bs, [batch]).
-
-to run all test suite modules starting with `bs` (i.e. all modules that test
-the bit syntax).
-
-To run a specific test case in a module, the full name of the module and test
-case must be specified:
-
-    ts:run(emulator, bs_bincomp_SUITE, byte_aligned, [batch]).
+```erlang
+ts:run(emulator, bs_construct_SUITE, [batch]).
+```
+
+to run the tests in the `bs_construct_SUITE` module (testing construction of
+binaries using the binary syntax).
+
+It is also possible to run a single test case by the specifying the module name and a function name:
+
+```erlang
+ts:run(emulator, bs_bincomp_SUITE, byte_aligned, [batch]).
+```
 
 Run `ts:help().` for more information.
 
 As of R14B02 it is also possibly to start all tests but the erl_interface tests
 by invoking Common Test directly from the released applications test directory,
 i.e.
-
-    cd $TESTROOT/test_server
-    $ERL_TOP/bin/ct_run -suite ../compiler_test/andor_SUITE -case t_orelse
+```bash
+cd $TESTROOT/test_server
+$ERL_TOP/bin/ct_run -suite ../compiler_test/andor_SUITE -case t_orelse
+```
 
 Running [ct_run][] from the command line still requires you to do the
 `ts:install()` step above.
 
-### Convenience for running tests without the release and configuration steps
-
-It can be convenient to run tests with a single command. This way, one
-do not need to worry about missing to run `make release_tests` after
-changing a test suite. The `make test` command can be used for this
-purpose. The `make test` command works when the current directory
-contains a directory called test and in the root directory of the
-source code tree.
-
-*(Waring)* Some test cases do not run correctly or cannot be run at
-all through the `make test` command (typically test cases that require
-test specific C code to be compiled) because `make test` runs tests
-directly by invoking the `ct_run` command instead of using the `ts`
-wrapper. One has to follow the procedure described above to run test
-cases that do not work with `make test`.
-
-Below are some examples that illustrate how `make test` can be
-used:
-
-    # ERL_TOP needs to be set correctly
-    cd /path/to/otp
-    export ERL_TOP=`pwd`
-
-    # Build Erlang/OTP
-    #
-    # Note that make test will only compile test code except when
-    # make test is executed from $ERL_TOP.
-    ./otp_build setup -a
-
-    # Run a test case (The ARGS variable is passed to ct_run)
-    (cd $ERL_TOP/erts/emulator && make ARGS="-suite binary_SUITE -case deep_bitstr_lists" test)
-
-    # Run a test suite
-    (cd $ERL_TOP/lib/stdlib && make ARGS="-suite ets_SUITE" test)
-
-    # Run all test suites for an application
-    (cd $ERL_TOP/lib/asn1 && make test)
-
-    # Run all tests
-    #
-    # When executed from $ERL_TOP, "make test" will first release and
-    # configure all tests and then attempt to run all tests with `ts:run`.
-    # This will take several hours.
-    (cd $ERL_TOP && make test)
-
-
 Examining the results
 ---------------------
 
-Open the file `release/tests/test_server/index.html` in a web browser. Or open
-`release/tests/test_server/last_test.html` when a test suite is running to
-examine the results so far for the currently executing test suite (in R14B02 and
-later you want to open the `release/tests/test_server/all_runs.html` file to
-get to the currently running test)
+Open the file `$ERL_TOP/release/tests/test_server/index.html` in a web browser. Or open
+`$ERL_TOP/release/tests/test_server/last_test.html` when a test suite is running to
+examine the results so far for the currently executing test suite.
 
    [ct_run]: http://www.erlang.org/doc/man/ct_run.html
    [ct hook]: http://www.erlang.org/doc/apps/common_test/ct_hooks_chapter.html
@@ -192,5 +202,6 @@ get to the currently running test)
    [common_test]: http://www.erlang.org/doc/man/ct.html
    [data_dir]: http://www.erlang.org/doc/apps/common_test/write_test_chapter.html#data_priv_dir
    [configuring the tests]: #configuring-the-test-environment
+   [Running tests while developing]: #running-tests-while-developing
 
    [?TOC]: true
Index: otp-OTP-22.2.7/Makefile.in
===================================================================
--- otp-OTP-22.2.7.orig/Makefile.in
+++ otp-OTP-22.2.7/Makefile.in
@@ -477,7 +477,14 @@ BOOTSTRAP_COMPILER  =  $(BOOTSTRAP_TOP)/
 # otp.mk is only used to figure out if we are doing PGO or not
 include $(ERL_TOP)/make/$(TARGET)/otp.mk
 
-.PHONY: emulator libs kernel stdlib compiler hipe syntax_tools preloaded
+.PHONY: emulator libs kernel stdlib compiler hipe syntax_tools preloaded system
+
+emulator_test: emulator
+	$(make_verbose)cd erts/emulator && ERL_TOP=$(ERL_TOP) TYPE=$(TYPE) $(MAKE) test
+epmd_test:
+	$(make_verbose)cd erts/epmd && ERL_TOP=$(ERL_TOP) TYPE=$(TYPE) $(MAKE) test
+system_test:
+	$(make_verbose)cd erts && ERL_TOP=$(ERL_TOP) TYPE=$(TYPE) $(MAKE) test
 
 ifeq ($(USE_PGO), true)
 PROFILE=use
@@ -1205,9 +1212,14 @@ bootstrap_clean:
 	$(V_at)test $(BOOTSTRAP_ROOT) = $(ERL_TOP) \
 		|| $(MAKE) BOOTSTRAP_ROOT=$(BOOTSTRAP_ROOT) bootstrap_root_clean
 
+APPS_TEST=$(patsubst %, %_test,$(APPS))
+.SECONDEXPANSION:
+$(APPS_TEST): $$(patsubst %_test,%,$$@)
+	ERL_TOP=$(ERL_TOP) TYPE=$(TYPE) $(MAKE) -C lib/$< test
 # ----------------------------------------------------------------------
 
 .PHONY: test
 
 test: all release release_tests
+	chmod +x $(ERL_TOP)/make/test_target_script.sh
 	$(ERL_TOP)/make/test_target_script.sh $(ERL_TOP)
Index: otp-OTP-22.2.7/README.md
===================================================================
--- otp-OTP-22.2.7.orig/README.md
+++ otp-OTP-22.2.7/README.md
@@ -68,6 +68,10 @@ Read our [contribution guide](CONTRIBUTI
 
 We have a list of [Help Wanted](https://bugs.erlang.org/issues/?jql=status%20%3D%20%22Help%20Wanted%22) bugs that we would appreciate external help from the community. This is a great place to get involved.
 
+## Awesome-Erlang
+
+You can find more projects, tools and articles related to Erlang/OTP on the [awesome-erlang list](https://github.com/drobakowski/awesome-erlang). Add your project there.
+
 ## License
 
 Erlang/OTP is released under the [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0).
@@ -89,6 +93,3 @@ Erlang/OTP is released under the [Apache
 > limitations under the License.
 >
 > %CopyrightEnd%
-
-## Awesome-Erlang
-You can find more projects, tools and articles related to Erlang/OTP on the [awesome-erlang list](https://github.com/drobakowski/awesome-erlang). Add your project there.
Index: otp-OTP-22.2.7/erts/Makefile
===================================================================
--- otp-OTP-22.2.7.orig/erts/Makefile
+++ otp-OTP-22.2.7/erts/Makefile
@@ -40,7 +40,7 @@ all: smp
 docs:
 	$(V_at)( cd doc/src && $(MAKE) $@ )
 
-.PHONY: debug opt lcnt clean
+.PHONY: debug opt lcnt clean test
 debug opt lcnt clean:
 	$(V_at)for d in emulator $(ERTSDIRS); do \
 		if test -d $$d; then \
@@ -149,9 +149,12 @@ endif
 .PHONY: release_docs
 release_docs:
 	$(V_at)( cd doc/src && $(MAKE) $@ )
+	
+.PHONY: test
+test:
+	TEST_NEEDS_RELEASE=true TYPE=$(TYPE) \
+	  $(ERL_TOP)/make/test_target_script.sh $(ERL_TOP)
 
 .PHONY: xmllint
 xmllint:
 	$(MAKE) -C doc/src $@
-
-include $(ERL_TOP)/make/app_targets.mk
Index: otp-OTP-22.2.7/lib/common_test/test_server/ts.erl
===================================================================
--- otp-OTP-22.2.7.orig/lib/common_test/test_server/ts.erl
+++ otp-OTP-22.2.7/lib/common_test/test_server/ts.erl
@@ -33,6 +33,7 @@
 	 estone/0, estone/1,
 	 cross_cover_analyse/1,
 	 compile_testcases/0, compile_testcases/1,
+         compile_datadirs/2,
 	 help/0]).
 
 %% Functions kept for backwards compatibility
@@ -1011,7 +1012,9 @@ compile_testcases([]) ->
     ok.
 
 compile_datadirs(DataDirs) ->
-    {ok,Variables} = file:consult("variables"),
+    compile_datadirs("variables", DataDirs).
+compile_datadirs(VariablesFile, DataDirs) ->
+    {ok,Variables} = file:consult(VariablesFile),
 
     lists:foreach(fun(Dir) ->
 			  ts_lib:make_non_erlang(Dir, Variables)
Index: otp-OTP-22.2.7/make/app_targets.mk
===================================================================
--- otp-OTP-22.2.7.orig/make/app_targets.mk
+++ otp-OTP-22.2.7/make/app_targets.mk
@@ -22,4 +22,6 @@
 .PHONY: test
 
 test:
-	$(ERL_TOP)/make/test_target_script.sh $(ERL_TOP)
+	chmod +x $(ERL_TOP)/make/test_target_script.sh
+	TEST_NEEDS_RELEASE=$(TEST_NEEDS_RELEASE) TYPE=$(TYPE) \
+	  $(ERL_TOP)/make/test_target_script.sh $(ERL_TOP)
Index: otp-OTP-22.2.7/make/test_target_script.sh
===================================================================
--- otp-OTP-22.2.7.orig/make/test_target_script.sh
+++ otp-OTP-22.2.7/make/test_target_script.sh
@@ -20,7 +20,6 @@
 # %CopyrightEnd%
 #
 
-
 RED='\033[0;31m'
 GREEN='\033[0;32m'
 YELLOW='\033[1;33m'
@@ -58,8 +57,13 @@ All tests will require several hours to
 following text file that describes how to run tests for a specific
 application.
 
+  $ERL_TOP/HOWTO/TESTING.md
+
+You can follow the test results here:
+
+  file://$ERL_TOP/release/tests/test_server/index.html
+
 EOM
-        echo $ERL_TOP/HOWTO/TESTING.md
     }
     print_highlighted_msg_with_printer $YELLOW print_msg
 }
@@ -90,25 +94,6 @@ EOM
     print_highlighted_msg_with_printer $LIGHT_CYAN print_msg
 }
 
-print_c_files_warning () {
-    print_msg () {
-        cat << EOM
-
-WARNING
-
-The test directory contains .c files which means that some test cases
-will probably not work correctly when run through "make test". The
-text file at the following location describes how one can compile and
-run all test cases:
-
-
-EOM
-        echo $ERL_TOP/HOWTO/TESTING.md
-    }
-    print_highlighted_msg_with_printer $YELLOW print_msg
-}
-
-
 print_on_error_note () {
     print_msg () {
         cat << EOM
@@ -128,6 +113,18 @@ EOM
     print_highlighted_msg_with_printer $NC print_msg
 }
 
+release_erlang () {
+    local RELEASE_ROOT=${1}
+    if ! (cd $ERL_TOP && make release RELEASE_ROOT="${RELEASE_ROOT}"); then
+        return 1
+    fi
+    if ! (cd "$RELEASE_ROOT" && ./Install -minimal "`pwd`"); then
+        return 1
+    fi
+    export PATH="${RELEASE_ROOT}/bin:$PATH"
+    return 0
+}
+
 # Check ERL_TOP
 
 if [ -d "$1" ]
@@ -150,7 +147,6 @@ fi
 
 export ERL_TOP=$ERL_TOP
 
-
 if [ -z "${ARGS}" ]
 then
    ARGS="$@"
@@ -160,7 +156,7 @@ fi
 DIR=`pwd`
 if [ "$DIR" -ef "$ERL_TOP" ]
 then
-    TARGET_SYS=`$ERL_TOP/erts/autoconf/config.guess`
+    TARGET_SYS=`cat $ERL_TOP/bootstrap/target`
     REL_DIR="$ERL_TOP/release/$TARGET_SYS"
     cd "$REL_DIR"
     ./Install -minimal "`pwd`"
@@ -182,15 +178,47 @@ then
     exit 1
 fi
 
-
 APPLICATION="`basename $DIR`"
+
+if [ "$APPLICATION" = "erts" ]; then
+    APPLICATION="system"
+fi
+
 CT_RUN="$ERL_TOP/bin/ct_run"
+PATH="${ERL_TOP}/bin/:${PATH}"
 MAKE_TEST_DIR="`pwd`/make_test_dir"
 MAKE_TEST_REL_DIR="$MAKE_TEST_DIR/${APPLICATION}_test"
 MAKE_TEST_CT_LOGS="$MAKE_TEST_DIR/ct_logs"
-RELEASE_TEST_SPEC_LOG="$MAKE_TEST_CT_LOGS/release_tests_spec_log"
+RELEASE_TEST_SPEC_LOG="$MAKE_TEST_DIR/release_tests_spec_log"
+INSTALL_TEST_LOG="$MAKE_TEST_DIR/install_tests_log"
+COMPILE_TEST_LOG="$MAKE_TEST_DIR/compile_tests_log"
+RELEASE_ROOT="${MAKE_TEST_DIR}/otp"
+RELEASE_LOG="$MAKE_TEST_DIR/release_tests_log"
 
 cd test
+
+mkdir -p "$MAKE_TEST_DIR"
+mkdir -p "$MAKE_TEST_CT_LOGS"
+
+# Check that we are running a released erlang when we have to
+if [ "$TEST_NEEDS_RELEASE" = "true" ]; then
+    MSG=$(cat <<EOF
+This application needs to be tested using a released Erlang/OTP.
+Erlang/OTP will now be released to ${RELEASE_ROOT}.
+EOF
+       )
+    print_highlighted_msg $YELLOW "${MSG}"
+    release_erlang "${RELEASE_ROOT}" > "${RELEASE_LOG}" 2>&1
+    if [ $? != 0 ]
+    then
+        print_highlighted_msg $RED "\"make release RELEASE_ROOT=${RELEASE_ROOT}\" failed.\nSee ${RELEASE_LOG} for full logs"
+        tail -30 "${RELEASE_LOG}"
+        exit 1
+    fi
+    CT_RUN="${RELEASE_ROOT}/bin/ct_run"
+    PATH=${RELEASE_ROOT}/bin/:${PATH}
+fi
+
 echo "The tests in test directory for $APPLICATION will be executed with ct_run"
 if [ -z "${ARGS}" ]
 then
@@ -204,31 +232,75 @@ then
     fi
 fi
 
-mkdir -p "$MAKE_TEST_DIR"
-mkdir -p "$MAKE_TEST_REL_DIR"
-mkdir -p "$MAKE_TEST_CT_LOGS"
-make RELSYSDIR=$MAKE_TEST_REL_DIR release_tests_spec > $RELEASE_TEST_SPEC_LOG 2>&1
+make RELEASE_PATH=$MAKE_TEST_DIR release_tests_spec > $RELEASE_TEST_SPEC_LOG 2>&1
 
 if [ $? != 0 ]
 then
     cat $RELEASE_TEST_SPEC_LOG
-    print_highlighted_msg $RED "\"make RELSYSDIR="$MAKE_TEST_REL_DIR" release_tests_spec\" failed."
+    print_highlighted_msg $RED "\"make RELEASE_PATH="$MAKE_TEST_DIR" release_tests_spec\" failed."
     exit 1
 fi
-SPEC_FLAG=""
-SPEC_FILE=""
+
 if [ -z "${ARGS}" ]
 then
-    SPEC_FLAG="-spec"
+    SPEC_FILE_POSTFIX="$MAKE_TEST_REL_DIR/${APPLICATION}_${SPEC_POSTFIX}.spec"
     SPEC_FILE="$MAKE_TEST_REL_DIR/$APPLICATION.spec"
-    ARGS="$SPEC_FLAG $SPEC_FILE"
+    if [ -f "$SPEC_FILE_POSTFIX" ]; then
+        SPEC_FILE="$SPEC_FILE_POSTFIX"
+   fi
+    ARGS="-spec $SPEC_FILE"
+fi
+
+ARGS="${ARGS} ${EXTRA_ARGS}"
+
+if ([ -n "${TYPE}" ] || [ -n "${FLAVOR}" ]) && [ "${WSLcross}" = "true" ]; then
+    print_highlighted_msg $RED "Setting TYPE or FLAVOR is not implemented yet for WSL"
+    exit 1;
+fi
+
+if [ -n "${TYPE}" ]; then
+    ERL_AFLAGS="${ERL_AFLAGS} -emu_type ${TYPE}"
 fi
 # Compile test server
-(cd "$ERL_TOP/lib/common_test/test_server" && make)
+if [ -n "${FLAVOR}" ]; then
+    ERL_AFLAGS="${ERL_AFLAGS} -emu_flavor ${FLAVOR}"
+fi
+
+# Compile test server and configure
+if [ ! -f "$ERL_TOP/lib/common_test/test_server/variables" ]; then
+    cd "$ERL_TOP/lib/common_test/test_server"
+    ( make && erl -noshell -eval "ts:install()." -s init stop )  > "$INSTALL_TEST_LOG" 2>&1
+    if [ $? != 0 ]
+    then
+        cat "$INSTALL_TEST_LOG"
+        print_highlighted_msg $RED "\"make && erl -eval 'ts:install()'\" in common_test/test_server failed."
+        exit 1
+    fi
+fi
+
+erl -sname test -noshell -pa "$ERL_TOP/lib/common_test/test_server" \
+    -eval "ts:compile_datadirs(\"$ERL_TOP/lib/common_test/test_server/variables\",\"*_SUITE_data\")."\
+    -s init stop > "$COMPILE_TEST_LOG" 2>&1
+
+if [ $? != 0 ]
+then
+    cat "$COMPILE_TEST_LOG"
+    print_highlighted_msg $RED "\"erl -eval 'ts:compile_datadirs/2'\" failed."
+    exit 1
+fi
+
+
 # Run ct_run
 cd $MAKE_TEST_REL_DIR
-$CT_RUN -logdir $MAKE_TEST_CT_LOGS\
+if [ -n "${CTRUN_TIMEOUT}" ]; then
+        CTRUN_TIMEOUT="timeout -s ABRT --foreground --preserve-status $((${CTRUN_TIMEOUT}+5))m timeout -s USR1 --foreground --preserve-status ${CTRUN_TIMEOUT}m"
+fi
+ERL_AFLAGS="${ERL_AFLAGS}" $CTRUN_TIMEOUT \
+      $CT_RUN -logdir $MAKE_TEST_CT_LOGS\
         -pa "$ERL_TOP/lib/common_test/test_server"\
+        -config "$ERL_TOP/lib/common_test/test_server/ts.config"\
+        -config "$ERL_TOP/lib/common_test/test_server/ts.unix.config"\
+        -exit_status ignore_config \
         ${ARGS}\
         -erl_args\
         -env "$PATH"\
Index: otp-OTP-22.2.7/make/target.mk
===================================================================
--- otp-OTP-22.2.7.orig/make/target.mk
+++ otp-OTP-22.2.7/make/target.mk
@@ -24,7 +24,7 @@ ifeq ($(OVERRIDE_TARGET),)
 
 ifeq ($(TARGET),)
 
-TARGET := $(shell $(ERL_TOP)/erts/autoconf/config.guess)
+TARGET := $(shell cat $(ERL_TOP)/bootstrap/target)
 
 else

Places

File feature-upstream-testsuite-enhancement-2.patch of Package erlang.30223

Places