diff options
Diffstat (limited to 'ffmpeg/doc/fate.html')
| -rw-r--r-- | ffmpeg/doc/fate.html | 345 |
1 files changed, 345 insertions, 0 deletions
diff --git a/ffmpeg/doc/fate.html b/ffmpeg/doc/fate.html new file mode 100644 index 0000000..fba60d1 --- /dev/null +++ b/ffmpeg/doc/fate.html @@ -0,0 +1,345 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> +<html> +<!-- Created by GNU Texinfo 6.8, https://www.gnu.org/software/texinfo/ --> + <head> + <meta charset="utf-8"> + <title> + FFmpeg Automated Testing Environment + </title> + <meta name="viewport" content="width=device-width,initial-scale=1.0"> + <link rel="stylesheet" type="text/css" href="bootstrap.min.css"> + <link rel="stylesheet" type="text/css" href="style.min.css"> + </head> + <body> + <div class="container"> + <h1> + FFmpeg Automated Testing Environment + </h1> +<div align="center"> +</div> + + +<a name="Top"></a> +<a name="SEC_Top"></a> + +<div class="Contents_element" id="SEC_Contents"> +<h2 class="contents-heading">Table of Contents</h2> + +<div class="contents"> + +<ul class="no-bullet"> + <li><a id="toc-Introduction" href="#Introduction">1 Introduction</a></li> + <li><a id="toc-Using-FATE-from-your-FFmpeg-source-directory" href="#Using-FATE-from-your-FFmpeg-source-directory">2 Using FATE from your FFmpeg source directory</a></li> + <li><a id="toc-Submitting-the-results-to-the-FFmpeg-result-aggregation-server" href="#Submitting-the-results-to-the-FFmpeg-result-aggregation-server">3 Submitting the results to the FFmpeg result aggregation server</a></li> + <li><a id="toc-Uploading-new-samples-to-the-fate-suite" href="#Uploading-new-samples-to-the-fate-suite">4 Uploading new samples to the fate suite</a></li> + <li><a id="toc-FATE-makefile-targets-and-variables" href="#FATE-makefile-targets-and-variables">5 FATE makefile targets and variables</a> + <ul class="no-bullet"> + <li><a id="toc-Makefile-targets" href="#Makefile-targets">5.1 Makefile targets</a></li> + <li><a id="toc-Makefile-variables" href="#Makefile-variables">5.2 Makefile variables</a></li> + <li><a id="toc-Examples" href="#Examples">5.3 Examples</a></li> + </ul></li> +</ul> +</div> +</div> + +<a name="Introduction"></a> +<h2 class="chapter">1 Introduction<span class="pull-right"><a class="anchor hidden-xs" href="#Introduction" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Introduction" aria-hidden="true">TOC</a></span></h2> + +<p>FATE is an extended regression suite on the client-side and a means +for results aggregation and presentation on the server-side. +</p> +<p>The first part of this document explains how you can use FATE from +your FFmpeg source directory to test your ffmpeg binary. The second +part describes how you can run FATE to submit the results to FFmpeg’s +FATE server. +</p> +<p>In any way you can have a look at the publicly viewable FATE results +by visiting this website: +</p> +<p><a href="http://fate.ffmpeg.org/">http://fate.ffmpeg.org/</a> +</p> +<p>This is especially recommended for all people contributing source +code to FFmpeg, as it can be seen if some test on some platform broke +with their recent contribution. This usually happens on the platforms +the developers could not test on. +</p> +<p>The second part of this document describes how you can run FATE to +submit your results to FFmpeg’s FATE server. If you want to submit your +results be sure to check that your combination of CPU, OS and compiler +is not already listed on the above mentioned website. +</p> +<p>In the third part you can find a comprehensive listing of FATE makefile +targets and variables. +</p> + +<a name="Using-FATE-from-your-FFmpeg-source-directory"></a> +<h2 class="chapter">2 Using FATE from your FFmpeg source directory<span class="pull-right"><a class="anchor hidden-xs" href="#Using-FATE-from-your-FFmpeg-source-directory" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Using-FATE-from-your-FFmpeg-source-directory" aria-hidden="true">TOC</a></span></h2> + +<p>If you want to run FATE on your machine you need to have the samples +in place. You can get the samples via the build target fate-rsync. +Use this command from the top-level source directory: +</p> +<div class="example"> +<pre class="example">make fate-rsync SAMPLES=fate-suite/ +make fate SAMPLES=fate-suite/ +</pre></div> + +<p>The above commands set the samples location by passing a makefile +variable via command line. It is also possible to set the samples +location at source configuration time by invoking configure with +<samp>--samples=<path to the samples directory></samp>. Afterwards you can +invoke the makefile targets without setting the <var>SAMPLES</var> makefile +variable. This is illustrated by the following commands: +</p> +<div class="example"> +<pre class="example">./configure --samples=fate-suite/ +make fate-rsync +make fate +</pre></div> + +<p>Yet another way to tell FATE about the location of the sample +directory is by making sure the environment variable FATE_SAMPLES +contains the path to your samples directory. This can be achieved +by e.g. putting that variable in your shell profile or by setting +it in your interactive session. +</p> +<div class="example"> +<pre class="example">FATE_SAMPLES=fate-suite/ make fate +</pre></div> + +<div class="info"> +<p>Do not put a ’~’ character in the samples path to indicate a home +directory. Because of shell nuances, this will cause FATE to fail. +</p></div> +<p>To get the complete list of tests, run the command: +</p><div class="example"> +<pre class="example">make fate-list +</pre></div> + +<p>You can specify a subset of tests to run by specifying the +corresponding elements from the list with the <code>fate-</code> prefix, +e.g. as in: +</p><div class="example"> +<pre class="example">make fate-ffprobe_compact fate-ffprobe_xml +</pre></div> + +<p>This makes it easier to run a few tests in case of failure without +running the complete test suite. +</p> +<p>To use a custom wrapper to run the test, pass <samp>--target-exec</samp> to +<code>configure</code> or set the <var>TARGET_EXEC</var> Make variable. +</p> + +<a name="Submitting-the-results-to-the-FFmpeg-result-aggregation-server"></a> +<h2 class="chapter">3 Submitting the results to the FFmpeg result aggregation server<span class="pull-right"><a class="anchor hidden-xs" href="#Submitting-the-results-to-the-FFmpeg-result-aggregation-server" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Submitting-the-results-to-the-FFmpeg-result-aggregation-server" aria-hidden="true">TOC</a></span></h2> + +<p>To submit your results to the server you should run fate through the +shell script <samp>tests/fate.sh</samp> from the FFmpeg sources. This script needs +to be invoked with a configuration file as its first argument. +</p> +<div class="example"> +<pre class="example">tests/fate.sh /path/to/fate_config +</pre></div> + +<p>A configuration file template with comments describing the individual +configuration variables can be found at <samp>doc/fate_config.sh.template</samp>. +</p> +<p>The mentioned configuration template is also available here: +</p><pre class="verbatim">slot= # some unique identifier +repo=git://source.ffmpeg.org/ffmpeg.git # the source repository +#branch=release/2.6 # the branch to test +samples= # path to samples directory +workdir= # directory in which to do all the work +#fate_recv="ssh -T fate@fate.ffmpeg.org" # command to submit report +comment= # optional description +build_only= # set to "yes" for a compile-only instance that skips tests +ignore_tests= + +# the following are optional and map to configure options +arch= +cpu= +cross_prefix= +as= +cc= +ld= +target_os= +sysroot= +target_exec= +target_path= +target_samples= +extra_cflags= +extra_ldflags= +extra_libs= +extra_conf= # extra configure options not covered above + +#make= # name of GNU make if not 'make' +makeopts= # extra options passed to 'make' +#makeopts_fate= # extra options passed to 'make' when running tests, + # defaulting to makeopts above if this is not set +#tar= # command to create a tar archive from its arguments on stdout, + # defaults to 'tar c' +</pre> +<p>Create a configuration that suits your needs, based on the configuration +template. The <code>slot</code> configuration variable can be any string that is not +yet used, but it is suggested that you name it adhering to the following +pattern ‘<samp><var>arch</var>-<var>os</var>-<var>compiler</var>-<var>compiler version</var></samp>’. The +configuration file itself will be sourced in a shell script, therefore all +shell features may be used. This enables you to setup the environment as you +need it for your build. +</p> +<p>For your first test runs the <code>fate_recv</code> variable should be empty or +commented out. This will run everything as normal except that it will omit +the submission of the results to the server. The following files should be +present in $workdir as specified in the configuration file: +</p> +<ul> +<li> configure.log + </li><li> compile.log + </li><li> test.log + </li><li> report + </li><li> version +</li></ul> + +<p>When you have everything working properly you can create an SSH key pair +and send the public key to the FATE server administrator who can be contacted +at the email address <a href="mailto:fate-admin@ffmpeg.org">fate-admin@ffmpeg.org</a>. +</p> +<p>Configure your SSH client to use public key authentication with that key +when connecting to the FATE server. Also do not forget to check the identity +of the server and to accept its host key. This can usually be achieved by +running your SSH client manually and killing it after you accepted the key. +The FATE server’s fingerprint is: +</p> +<dl compact="compact"> +<dt><span>‘<samp>RSA</samp>’</span></dt> +<dd><p>d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51 +</p></dd> +<dt><span>‘<samp>ECDSA</samp>’</span></dt> +<dd><p>76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86 +</p></dd> +</dl> + +<p>If you have problems connecting to the FATE server, it may help to try out +the <code>ssh</code> command with one or more <samp>-v</samp> options. You should +get detailed output concerning your SSH configuration and the authentication +process. +</p> +<p>The only thing left is to automate the execution of the fate.sh script and +the synchronisation of the samples directory. +</p> +<a name="Uploading-new-samples-to-the-fate-suite"></a> +<h2 class="chapter">4 Uploading new samples to the fate suite<span class="pull-right"><a class="anchor hidden-xs" href="#Uploading-new-samples-to-the-fate-suite" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Uploading-new-samples-to-the-fate-suite" aria-hidden="true">TOC</a></span></h2> + +<p>If you need a sample uploaded send a mail to samples-request. +</p> +<p>This is for developers who have an account on the fate suite server. +If you upload new samples, please make sure they are as small as possible, +space on each client, network bandwidth and so on benefit from smaller test cases. +Also keep in mind older checkouts use existing sample files, that means in +practice generally do not replace, remove or overwrite files as it likely would +break older checkouts or releases. +Also all needed samples for a commit should be uploaded, ideally 24 +hours, before the push. +If you need an account for frequently uploading samples or you wish to help +others by doing that send a mail to ffmpeg-devel. +</p> +<div class="example"> +<pre class="example">#First update your local samples copy: +rsync -vauL --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X fate-suite.ffmpeg.org:/home/samples/fate-suite/ ~/fate-suite + +#Then do a dry run checking what would be uploaded: +rsync -vanL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite + +#Upload the files: +rsync -vaL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite +</pre></div> + + +<a name="FATE-makefile-targets-and-variables"></a> +<h2 class="chapter">5 FATE makefile targets and variables<span class="pull-right"><a class="anchor hidden-xs" href="#FATE-makefile-targets-and-variables" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-FATE-makefile-targets-and-variables" aria-hidden="true">TOC</a></span></h2> + +<a name="Makefile-targets"></a> +<h3 class="section">5.1 Makefile targets<span class="pull-right"><a class="anchor hidden-xs" href="#Makefile-targets" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Makefile-targets" aria-hidden="true">TOC</a></span></h3> + +<dl compact="compact"> +<dt><span><samp>fate-rsync</samp></span></dt> +<dd><p>Download/synchronize sample files to the configured samples directory. +</p> +</dd> +<dt><span><samp>fate-list</samp></span></dt> +<dd><p>Will list all fate/regression test targets. +</p> +</dd> +<dt><span><samp>fate</samp></span></dt> +<dd><p>Run the FATE test suite (requires the fate-suite dataset). +</p></dd> +</dl> + +<a name="Makefile-variables"></a> +<h3 class="section">5.2 Makefile variables<span class="pull-right"><a class="anchor hidden-xs" href="#Makefile-variables" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Makefile-variables" aria-hidden="true">TOC</a></span></h3> + +<dl compact="compact"> +<dt><span><code>V</code></span></dt> +<dd><p>Verbosity level, can be set to 0, 1 or 2. + </p><ul> +<li> 0: show just the test arguments + </li><li> 1: show just the command used in the test + </li><li> 2: show everything + </li></ul> + +</dd> +<dt><span><code>SAMPLES</code></span></dt> +<dd><p>Specify or override the path to the FATE samples at make time, it has a +meaning only while running the regression tests. +</p> +</dd> +<dt><span><code>THREADS</code></span></dt> +<dd><p>Specify how many threads to use while running regression tests, it is +quite useful to detect thread-related regressions. +</p> +</dd> +<dt><span><code>THREAD_TYPE</code></span></dt> +<dd><p>Specify which threading strategy test, either ‘<samp>slice</samp>’ or ‘<samp>frame</samp>’, +by default ‘<samp>slice+frame</samp>’ +</p> +</dd> +<dt><span><code>CPUFLAGS</code></span></dt> +<dd><p>Specify CPU flags. +</p> +</dd> +<dt><span><code>TARGET_EXEC</code></span></dt> +<dd><p>Specify or override the wrapper used to run the tests. +The <code>TARGET_EXEC</code> option provides a way to run FATE wrapped in +<code>valgrind</code>, <code>qemu-user</code> or <code>wine</code> or on remote targets +through <code>ssh</code>. +</p> +</dd> +<dt><span><code>GEN</code></span></dt> +<dd><p>Set to ‘<samp>1</samp>’ to generate the missing or mismatched references. +</p> +</dd> +<dt><span><code>HWACCEL</code></span></dt> +<dd><p>Specify which hardware acceleration to use while running regression tests, +by default ‘<samp>none</samp>’ is used. +</p> +</dd> +<dt><span><code>KEEP</code></span></dt> +<dd><p>Set to ‘<samp>1</samp>’ to keep temp files generated by fate test(s) when test is successful. +Default is ‘<samp>0</samp>’, which removes these files. Files are always kept when a test +fails. +</p> +</dd> +</dl> + +<a name="Examples"></a> +<h3 class="section">5.3 Examples<span class="pull-right"><a class="anchor hidden-xs" href="#Examples" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Examples" aria-hidden="true">TOC</a></span></h3> + +<div class="example"> +<pre class="example">make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate +</pre></div> + <p style="font-size: small;"> + This document was generated using <a href="https://www.gnu.org/software/texinfo/"><em>makeinfo</em></a>. + </p> + </div> + </body> +</html> |