FATE is an extended regression suite on the client-side and a means +for results aggregation and presentation on the server-side. +
+The first part of this document explains how you can use FATE from +your FFmpeg source directory to test your ffmpeg binary. The second +part describes how you can run FATE to submit the results to FFmpeg’s +FATE server. +
+In any way you can have a look at the publicly viewable FATE results +by visiting this website: +
+ +This is especially recommended for all people contributing source +code to FFmpeg, as it can be seen if some test on some platform broke +with their recent contribution. This usually happens on the platforms +the developers could not test on. +
+The second part of this document describes how you can run FATE to +submit your results to FFmpeg’s FATE server. If you want to submit your +results be sure to check that your combination of CPU, OS and compiler +is not already listed on the above mentioned website. +
+In the third part you can find a comprehensive listing of FATE makefile +targets and variables. +
+ + +If you want to run FATE on your machine you need to have the samples +in place. You can get the samples via the build target fate-rsync. +Use this command from the top-level source directory: +
+make fate-rsync SAMPLES=fate-suite/ +make fate SAMPLES=fate-suite/ +
The above commands set the samples location by passing a makefile +variable via command line. It is also possible to set the samples +location at source configuration time by invoking configure with +--samples=<path to the samples directory>. Afterwards you can +invoke the makefile targets without setting the SAMPLES makefile +variable. This is illustrated by the following commands: +
+./configure --samples=fate-suite/ +make fate-rsync +make fate +
Yet another way to tell FATE about the location of the sample +directory is by making sure the environment variable FATE_SAMPLES +contains the path to your samples directory. This can be achieved +by e.g. putting that variable in your shell profile or by setting +it in your interactive session. +
+FATE_SAMPLES=fate-suite/ make fate +
Do not put a ’~’ character in the samples path to indicate a home +directory. Because of shell nuances, this will cause FATE to fail. +
To get the complete list of tests, run the command: +
make fate-list +
You can specify a subset of tests to run by specifying the
+corresponding elements from the list with the fate- prefix,
+e.g. as in:
+
make fate-ffprobe_compact fate-ffprobe_xml +
This makes it easier to run a few tests in case of failure without +running the complete test suite. +
+To use a custom wrapper to run the test, pass --target-exec to
+configure or set the TARGET_EXEC Make variable.
+
To submit your results to the server you should run fate through the +shell script tests/fate.sh from the FFmpeg sources. This script needs +to be invoked with a configuration file as its first argument. +
+tests/fate.sh /path/to/fate_config +
A configuration file template with comments describing the individual +configuration variables can be found at doc/fate_config.sh.template. +
+The mentioned configuration template is also available here: +
slot= # some unique identifier +repo=git://source.ffmpeg.org/ffmpeg.git # the source repository +#branch=release/2.6 # the branch to test +samples= # path to samples directory +workdir= # directory in which to do all the work +#fate_recv="ssh -T fate@fate.ffmpeg.org" # command to submit report +comment= # optional description +build_only= # set to "yes" for a compile-only instance that skips tests +ignore_tests= + +# the following are optional and map to configure options +arch= +cpu= +cross_prefix= +as= +cc= +ld= +target_os= +sysroot= +target_exec= +target_path= +target_samples= +extra_cflags= +extra_ldflags= +extra_libs= +extra_conf= # extra configure options not covered above + +#make= # name of GNU make if not 'make' +makeopts= # extra options passed to 'make' +#makeopts_fate= # extra options passed to 'make' when running tests, + # defaulting to makeopts above if this is not set +#tar= # command to create a tar archive from its arguments on stdout, + # defaults to 'tar c' ++
Create a configuration that suits your needs, based on the configuration
+template. The slot configuration variable can be any string that is not
+yet used, but it is suggested that you name it adhering to the following
+pattern ‘arch-os-compiler-compiler version’. The
+configuration file itself will be sourced in a shell script, therefore all
+shell features may be used. This enables you to setup the environment as you
+need it for your build.
+
For your first test runs the fate_recv variable should be empty or
+commented out. This will run everything as normal except that it will omit
+the submission of the results to the server. The following files should be
+present in $workdir as specified in the configuration file:
+
When you have everything working properly you can create an SSH key pair +and send the public key to the FATE server administrator who can be contacted +at the email address fate-admin@ffmpeg.org. +
+Configure your SSH client to use public key authentication with that key +when connecting to the FATE server. Also do not forget to check the identity +of the server and to accept its host key. This can usually be achieved by +running your SSH client manually and killing it after you accepted the key. +The FATE server’s fingerprint is: +
+d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51 +
76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86 +
If you have problems connecting to the FATE server, it may help to try out
+the ssh command with one or more -v options. You should
+get detailed output concerning your SSH configuration and the authentication
+process.
+
The only thing left is to automate the execution of the fate.sh script and +the synchronisation of the samples directory. +
+ +If you need a sample uploaded send a mail to samples-request. +
+This is for developers who have an account on the fate suite server. +If you upload new samples, please make sure they are as small as possible, +space on each client, network bandwidth and so on benefit from smaller test cases. +Also keep in mind older checkouts use existing sample files, that means in +practice generally do not replace, remove or overwrite files as it likely would +break older checkouts or releases. +Also all needed samples for a commit should be uploaded, ideally 24 +hours, before the push. +If you need an account for frequently uploading samples or you wish to help +others by doing that send a mail to ffmpeg-devel. +
+#First update your local samples copy: +rsync -vauL --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X fate-suite.ffmpeg.org:/home/samples/fate-suite/ ~/fate-suite + +#Then do a dry run checking what would be uploaded: +rsync -vanL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite + +#Upload the files: +rsync -vaL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite +
Download/synchronize sample files to the configured samples directory. +
+Will list all fate/regression test targets. +
+Run the FATE test suite (requires the fate-suite dataset). +
VVerbosity level, can be set to 0, 1 or 2. +
SAMPLESSpecify or override the path to the FATE samples at make time, it has a +meaning only while running the regression tests. +
+THREADSSpecify how many threads to use while running regression tests, it is +quite useful to detect thread-related regressions. +
+THREAD_TYPESpecify which threading strategy test, either ‘slice’ or ‘frame’, +by default ‘slice+frame’ +
+CPUFLAGSSpecify CPU flags. +
+TARGET_EXECSpecify or override the wrapper used to run the tests.
+The TARGET_EXEC option provides a way to run FATE wrapped in
+valgrind, qemu-user or wine or on remote targets
+through ssh.
+
GENSet to ‘1’ to generate the missing or mismatched references. +
+HWACCELSpecify which hardware acceleration to use while running regression tests, +by default ‘none’ is used. +
+KEEPSet to ‘1’ to keep temp files generated by fate test(s) when test is successful. +Default is ‘0’, which removes these files. Files are always kept when a test +fails. +
+make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate +
+ This document was generated using makeinfo. +
+