CERN Accelerating science

Blog

Extracting coverage information for FTS3

It took a while, but finally I have got the coverage information showing up in our Sonarqube instance (only accessible within CERN network).

Now we have the coverage information for FTS3, FTS3 REST API and GFAL2.

Extracting the coverage information for the Python code was fairly easy: just run nosetests with the option --with-coverage --cover-xml --cover-xml-file=coverage.xml and there you go. Of course, you need to configure the sonar-project.properties properly.

sonar.core.codeCoveragePlugin=cobertura

sonar.python.xunit.reportPath=nosetests.xml
sonar.python.coverage.reportPath=coverage.xml

However, extracting the coverage information for the C/C++ code (both for FTS3 and GFAL2) was a little bit trickier, specially for FTS3, since we need to run the server side compiled with the coverage instrumentation.

Basically, these were the steps:

Compile with coverage instrumentation

Pice of cake. Just compile as follows:

CFLAGS=--coverage CXXFLAGS=--coverage cmake "${SOURCE_DIR}" \
    -DMAINBUILD=ON -DSERVERBUILD=ON -DCLIENTBUILD=ON -DMYSQLBUILD=ON \
    -DTESTBUILD=ON
make -j2

With that, pretty much done for GFAL2. After the build, run ctest and you have the coverage data (more on how to send this coverage data to Sonarqube later). However, FTS3 must be started as a server before we get integration coverage.

Build inside a mock environment

Why? Because we need to install a bunch of dependencies without polluting the build machine.

Run the server inside the mock environment

The FTS3 code assumed several things that made running the compiled code just for the tests harder. For instance:

  1. Always switches to user and group fts3. It requires, of course, the user and group fts3 to exists on the node, which is not true for the CI machines. Also, if you build as root and then switch to fts3, the coverage data can not be collected.
  2. SOAP interface always run, which requires permissions to bind, a valid certificate...
    1. Eventually this could be done, but we are deprecating anyway
  3. Assumes log directory location, permissions,...

 

So I had to modify several bits on the code to make it easier to run FTS3 in this limited environment.

Run the tests

There is an external REST host listening for the requests that the test instance will run, since they share the DB. Jobs are submitted there, and picked by the instrumented FTS3.

Shut down the services

Sound easy, right? It wasn't. In order to store the coverage data, the process needs to terminate normally (this is, exit being called). So far so good. But, oh my, FTS3 tended to use _exit instead, which will *not* call cleanup and destructors, so no coverage data.

Bummer. Well, sed s/_exit/exit/g and it should be good, shouldn't it? Nope. It turned out that if you called exit, then destructors are called, which is what we wanted. But that triggered a lot of race conditions on the tear down of the services. Meaning, segfaults. Meaning, no coverage either.

So I had to go through a lot of debugging until I managed to get rid of these race conditions, and get a nicely ordered shut down of the service. With this, the coverage data is generated.

Colllect the coverage

Once here, we have a bunch of .gcda files which we need to aggregate. We use lcov first to aggregate into an .info, and lcov_covertura.py to generate an XML that can be parsed by, for instance, Jenkins... but not by the Sonarqube Community CXX Plugin.

Luckily, that was easy. Just use an existing XSLT that transforms one into the other, and good to go. This is how it looks on the sonar-project.properties file

sonar.core.codeCoveragePlugin=cobertura

sonar.cxx.cppcheck.reportPath=cppcheck.xml
sonar.cxx.rats.reportPath=rats.xml
sonar.cxx.vera.reportPath=vera.xml
sonar.cxx.pclint.reportPath=pclint.xml
sonar.cxx.compiler.parser=gcc
sonar.cxx.compiler.reportPath=build.log
sonar.cxx.coverage.reportPath=coverage-unit.xml
sonar.cxx.coverage.itReportPath=coverage-integration.xml
sonar.cxx.coverage.overallReportPath=coverage-overall.xml
sonar.cxx.xunit.reportPath=tests.xml
sonar.cxx.xunit.xsltURL=https://raw.githubusercontent.com/SonarOpenCommunity/sonar-cxx/master/sonar-cxx-plugin/src/main/resources/xsl/boosttest-1.x-to-junit-1.0.xsl

Putting all together

Four scripts put everything together so it can be easily run: coverage.sh, coverage-unit.sh, coverage-integration.sh and coverage-overall.sh. An additional one orchestrates the whole process in Jenkins.

 

SOAP Deprecation

We would like to announce our intent to deprecate the FTS3 SOAP API.

The SOAP API was initially in the project mainly to keep backwards compatibility towards the gLite FTS clients, and for quite some time they haven't been supported anymore.

The reasons for this deprecation are:

  • It is hard to extend. Any new features may require even the addition of new calls, depending on the original WSDL definitions.
  • It runs on the FTS3 core process, which also schedules and runs the transfers. Bugs and inestabilities in a user-facing code can potentially cause a crash on the service.
  • Experiments that have migrated fully to FTS3, and using its new features, have done so talking directly to the REST API, so it is a waste of resources to keep two entry points.

 

We will, of course, keep support for the command line tools, except the configuration ones. They are already partially able to talk REST, and they will become fully REST compliant soon. Additionally, to make things easier, it won't be even be necessary to manually use the switch "--rest". The CLI will be able to figure it out by itself.

Stages

  1. Beginning of 1Q 2016
    • Fully implement all the functionality for fts-transfer-* and fts-delegation-init in REST (FTS-286), and remove the need for "--rest"
    • Print a warning when SOAP is used, but keep working
    • Print a deprecation warning for the SOAP Python bindings (FTS-285, DONE)
    • Document alternatives for the fts-config family (FTS-387, DONE)
  2. Beginning of 3Q 2016
    • Remove SOAP support from the new clients
    • Remove the SOAP Python bindings
  3. Q1 2017
    • Remove SOAP support from the server side

Please, mind the timing is just a proposal and can be moved.

Actions to be taken by the users and experiments

  1. If you are using the REST API directly, you don't need to worry
  2. If you are using the CLI, as soon as FTS-286 is released, switch from port :8443 to :8446 (the CLI will remind you to do so)
  3. If you are using the gLite clients, move to the FTS3 clients or use the REST interface directly

Actions to be taken by system administrators and users with configuration rights

We intend to stop supporting fts-config and family, since they are seldom used, and only by a handful of people.

Everything you can do with them, you can already achieve using directly REST. We have an equivalence guide between the config CLI and the REST API.

Do not hesitate to let us know if you miss something in the documentation, or if the lack of configuration CLI can cause a big trouble.

 

Tags: 

We have migrated out documentation to a Git repository, from which online HTML pages are generated using GitBook automatically every night.

Keeping the documentation in Git allows us to keep history, and tag the documentation together with the software they document. We believe this helps to keep track of how the service has changed with time. Also, documentation for older versions will remain easily reachable in case someone needs them.

Here is the link to the new docs: http://fts3-docs.web.cern.ch/fts3-docs/

Pull requests are welcome! :)

New experimental feature: self-documenting API

Now the develop branch of the FTS3 RESTful API implements the auto-generation of documentation using the Swagger format, which is based on JSON.

The advantage of using this specific format is the possibility of using the Swagger UI, which allows to render more nicely the documentation.

You can see it in action in http://aalvarez.web.cern.ch/aalvarez/fts3api/

Mind that this is an experimental feature, and, additionally, the built-in documentation needs to be improved.

First functional prototype of WebFTS

Today Andres has presented the first working prototype of WebFTS. This web interface allows user to easily interact with FTS3 without the need for special command line clients. Only a web browser is needed.

This is a extremely early stage prototype, but it even includes the possibility for listing source and destination directories, which relies on the small subset of data management capabilities of our RESTful API.

Looks promising!