CERN Accelerating science

All releases

3.2.26

*Many REST interface optimizations
*Bulk files staging
*Per VO retry global mechanism
*Many web monitoring performance improvements and new features
*Use Exponential Moving Average (EMA) along with weighted-average to avoid frequent "slopes" in the auto-tuning algorithm
*Crash fixed with concurrent delegations
*Log ip of the users in the log file
*Support multi-op and multiple replica jobs using glite/fts3 clients
*Remove weak ciphers from openssl
*fts-transfer-list to include source_se, dest_se in the params list
*Specify max active for a given link, will vary between 2-MAX    
*Many MySql optimizations

 

-Apply the database changes below

SET storage_engine=INNODB;

ALTER TABLE t_optimize_active ADD ema   DOUBLE DEFAULT 0;

ALTER TABLE t_optimizer_evolution ADD agrthroughput   FLOAT DEFAULT NULL;

CREATE INDEX t_optimize_active_datetime  ON t_optimize_active(datetime); 

DROP TABLE t_stage_req;

CREATE TABLE `t_stage_req` (
  `vo_name` varchar(100) NOT NULL,
  `host` varchar(150) NOT NULL,
  `operation` varchar(150) NOT NULL,
  `concurrent_ops` int(11) DEFAULT '0',
  PRIMARY KEY (`vo_name`,`host`,`operation`)
);

CREATE TABLE `t_turl` (

  `source_surl` varchar(150) NOT NULL,
  `destin_surl` varchar(150) NOT NULL,
  `source_turl` varchar(150) NOT NULL,
  `destin_turl` varchar(150) NOT NULL,
  `datetime` timestamp NULL DEFAULT NULL,
  `throughput` float DEFAULT NULL,
  `finish` double DEFAULT '0',
  `fail` double DEFAULT '0',
  PRIMARY KEY (`source_surl`,`destin_surl`,`source_turl`,`destin_turl`),
  KEY `t_url_datetime` (`datetime`),
  KEY `t_url_finish` (`finish`),
  KEY `t_url_fail` (`fail`)
);

ALTER TABLE t_server_config ADD vo_name VARCHAR(100);

DROP TABLE t_dm;

CREATE TABLE `t_dm` (
  `file_id` int(11) NOT NULL AUTO_INCREMENT,
  `job_id` char(36) NOT NULL,
  `file_state` varchar(32) NOT NULL,
  `dmHost` varchar(150) DEFAULT NULL,
  `source_surl` varchar(900) DEFAULT NULL,
  `dest_surl` varchar(900) DEFAULT NULL,
  `source_se` varchar(150) DEFAULT NULL,
  `dest_se` varchar(150) DEFAULT NULL,
  `reason` varchar(2048) DEFAULT NULL,
  `checksum` varchar(100) DEFAULT NULL,
  `finish_time` timestamp NULL DEFAULT NULL,
  `start_time` timestamp NULL DEFAULT NULL,
  `job_finished` timestamp NULL DEFAULT NULL,
  `tx_duration` double DEFAULT NULL,
  `retry` int(11) DEFAULT '0',
  `user_filesize` double DEFAULT NULL,
  `file_metadata` varchar(1024) DEFAULT NULL,
  `activity` varchar(255) DEFAULT 'default',
  `dm_token` varchar(255) DEFAULT NULL,
  `retry_timestamp` timestamp NULL DEFAULT NULL,
  `wait_timestamp` timestamp NULL DEFAULT NULL,
  `wait_timeout` int(11) DEFAULT NULL,
  `hashed_id` int(10) unsigned DEFAULT '0',
  `vo_name` varchar(100) DEFAULT NULL,
  PRIMARY KEY (`file_id`),
  KEY `dm_job_id` (`job_id`),
  KEY `t_dm_all` (`vo_name`,`source_se`,`file_state`),
  KEY `t_dm_source_Se` (`source_se`,`file_state`),
  KEY `t_dm_state` (`file_state`,`hashed_id`),
  CONSTRAINT `t_dm_ibfk_1` FOREIGN KEY (`job_id`) REFERENCES `t_job` (`job_id`)
);

CREATE TABLE t_dm_backup AS (SELECT * FROM t_dm);

-Update all packages below

fts-*

gfal2-*

srm-ifce

CGSI-gSOAP

davix-*

-Restart services

fts-bringonline

fts-msg-bulk

fts-records-cleaner

fts-server

WebFTS New release 2.0.0

Bug

  • [FTS-126] - Refresh Button is not working for Dropbox
  • [FTS-127] - SelectAll does not activate transfers
  • [FTS-128] - Filter->Hide folders does not work for Dropbox
  • [FTS-130] - SE Autocompletion is broken

Improvement

  • [FTS-45] - Keep previous state while browsing between tabs
  • [FTS-86] - keep expanded job details when refreshing 'My Jobs'

New Feature

  • [FTS-37] - Suggest possible VOs to the user when delegating
  • [FTS-92] - Warning for about to expire proxy

Task

  • [FTS-50] - Create the interface for Dropbox-like storages
  • [FTS-89] - Multihop job cannot report info on source_se and dest_se
  • [FTS-90] - WebFTS: keep transfer options in case of resubmission
  • [FTS-112] - Integrate the new FTS REST call to retrieve the latest X ( configurable) jobs
  • [FTS-115] - merge CloudStorage branch
  • [FTS-116] - WebFTS packaging
  • [FTS-125] - Merge site tour, new landing page and all Christina work
  • [FTS-129] - WebFTS configuration file
  • [FTS-131] - Create "official" webfts app for dropbox
  • [FTS-132] - Include News Feed on the landing page

FTS3 new release 3.2.26

 

*Many REST interface optimizations
*Bulk files staging
*Per VO retry global mechanism
*Many web monitoring performance improvements and new features
*Use Exponential Moving Average (EMA) along with weighted-average to avoid frequent "slopes" in the auto-tuning algorithm
*Crash fixed with concurrent delegations
*Log ip of the users in the log file
*Support multi-op and multiple replica jobs using glite/fts3 clients
*Remove weak ciphers from openssl
*fts-transfer-list to include source_se, dest_se in the params list
*Specify max active for a given link, will vary between 2-MAX    
*Many MySql optimizations

 

-Apply the database changes below

SET storage_engine=INNODB;

ALTER TABLE t_optimize_active ADD ema   DOUBLE DEFAULT 0;

ALTER TABLE t_optimizer_evolution ADD agrthroughput   FLOAT DEFAULT NULL;

CREATE INDEX t_optimize_active_datetime  ON t_optimize_active(datetime); 

DROP TABLE t_stage_req;

CREATE TABLE `t_stage_req` (
  `vo_name` varchar(100) NOT NULL,
  `host` varchar(150) NOT NULL,
  `operation` varchar(150) NOT NULL,
  `concurrent_ops` int(11) DEFAULT '0',
  PRIMARY KEY (`vo_name`,`host`,`operation`)
);

CREATE TABLE `t_turl` (

  `source_surl` varchar(150) NOT NULL,
  `destin_surl` varchar(150) NOT NULL,
  `source_turl` varchar(150) NOT NULL,
  `destin_turl` varchar(150) NOT NULL,
  `datetime` timestamp NULL DEFAULT NULL,
  `throughput` float DEFAULT NULL,
  `finish` double DEFAULT '0',
  `fail` double DEFAULT '0',
  PRIMARY KEY (`source_surl`,`destin_surl`,`source_turl`,`destin_turl`),
  KEY `t_url_datetime` (`datetime`),
  KEY `t_url_finish` (`finish`),
  KEY `t_url_fail` (`fail`)
);

ALTER TABLE t_server_config ADD vo_name VARCHAR(100);

DROP TABLE t_dm;

CREATE TABLE `t_dm` (
  `file_id` int(11) NOT NULL AUTO_INCREMENT,
  `job_id` char(36) NOT NULL,
  `file_state` varchar(32) NOT NULL,
  `dmHost` varchar(150) DEFAULT NULL,
  `source_surl` varchar(900) DEFAULT NULL,
  `dest_surl` varchar(900) DEFAULT NULL,
  `source_se` varchar(150) DEFAULT NULL,
  `dest_se` varchar(150) DEFAULT NULL,
  `reason` varchar(2048) DEFAULT NULL,
  `checksum` varchar(100) DEFAULT NULL,
  `finish_time` timestamp NULL DEFAULT NULL,
  `start_time` timestamp NULL DEFAULT NULL,
  `job_finished` timestamp NULL DEFAULT NULL,
  `tx_duration` double DEFAULT NULL,
  `retry` int(11) DEFAULT '0',
  `user_filesize` double DEFAULT NULL,
  `file_metadata` varchar(1024) DEFAULT NULL,
  `activity` varchar(255) DEFAULT 'default',
  `dm_token` varchar(255) DEFAULT NULL,
  `retry_timestamp` timestamp NULL DEFAULT NULL,
  `wait_timestamp` timestamp NULL DEFAULT NULL,
  `wait_timeout` int(11) DEFAULT NULL,
  `hashed_id` int(10) unsigned DEFAULT '0',
  `vo_name` varchar(100) DEFAULT NULL,
  PRIMARY KEY (`file_id`),
  KEY `dm_job_id` (`job_id`),
  KEY `t_dm_all` (`vo_name`,`source_se`,`file_state`),
  KEY `t_dm_source_Se` (`source_se`,`file_state`),
  KEY `t_dm_state` (`file_state`,`hashed_id`),
  CONSTRAINT `t_dm_ibfk_1` FOREIGN KEY (`job_id`) REFERENCES `t_job` (`job_id`)
);

CREATE TABLE t_dm_backup AS (SELECT * FROM t_dm);

-Update all packages below

fts-*

gfal2-*

srm-ifce

CGSI-gSOAP

davix-*

-Restart services

fts-bringonline

fts-msg-bulk

fts-records-cleaner

fts-server

 

FTS3 REST 3.2.3 tagged

Bug

  • [FTS-30] - Validate that the proxy is properly signed
  • [FTS-35] - When listing a davs:// endpoint, delegation stop working
  • [FTS-60] - WebDAV third party copy delegation is broken
  • [FTS-67] - REST CLI does not send properly formed json metadata

Improvement

  • [FTS-27] - Answer with 419 Authentication Timeout when a new delegation is needed
  • [FTS-31] - Use random distribution for hashed_id
  • [FTS-33] - In submission, insert OptimizerActive only for unique (source_se, dest_se)
  • [FTS-34] - In submission, use bulk inserts
  • [FTS-36] - Delegation id must be generated using all fqans
  • [FTS-66] - REST CLI doesn't work in SL5
  • [FTS-76] - Improve rest client error handling

New Feature

  • [FTS-24] - VOMS proxy support
  • [FTS-69] - REST is missing multihop submissions

Task

  • [FTS-26] - Improve logging for REST
  • [FTS-29] - selinux rpm for test must let apache run voms-proxy-init and create the proxy
  • [FTS-32] - Check queries being generated by SqlAlchemy
  • [FTS-75] - REST: Provide reference Python functions for LHCb

User documentation

  • [FTS-25] - Improve REST API documentation

FTS3 announcement of updates

FTS3 updates will be announced in:

 

FTS3 releases explanation

Production release:

  • The version currently installed at CERN and used by the experiments for production transfers

Release candidate:

  • The version installed in the pilot (passed all certification tests in the test-bed infrastructure)

EPEL release:

  • EPEL is the formal repository for FTS3 stable releases, basically what's in EPEL is in production

Development version:

  • The version which is installed and testing is ongoing in the development test-bed, next step to move to the pilot service

 

3.1.74

  • reduce send/recv timeout to 110 secs to prevent from killing the clients ungracefully
  • leaking file descriptors when gsoap is set to keep_alive mode
  • monitoring web-app efficiency plots and optimizer decisions
  • bug-fix for reporting transfer duration using *-status CLI
  • do not use decimal digits for efficiency calculation
  • bug-fix for enabling logging in debug mode
  • prevent a MySql? deadlock and inserting records into retry tables

 

Important notes:

You will need to manually create the columns below:

 

  ALTER TABLE t_optimize_active ADD datetime TIMESTAMP  NULL DEFAULT NULL;
  ALTER TABLE t_optimize_active ADD message VARCHAR(512);
  ALTER TABLE t_file ADD activity VARCHAR(255) DEFAULT "default";
  ALTER TABLE t_optimize ADD udt VARCHAR(3) DEFAULT 'off'

 

3.2.22

  • UDT support, configuring FTS3 to use GridFTP/UDT instead of GridFTP/TCP (will fallback automatically to TCP if UDT fails )

  • fix for parsing MySql? Db connection string when contains IPv6 address

  • global transfer timeout can now be set, along with complimentary seconds per megabyte (fts-config-set CLI)

  • improved error messages, clearly state the source of failure and the phase (SOURCE, DESTINATION, etc)

  • introduce max limits for number of active of a given source or destination endpoint (if set will vary between 2-MAX)

  • apply bandwidth restrictions to individual endpoints served as source or destination of a given link

  • persistently drain hosts to survive restarts, draining state is now stored into the db

  • optimize MySql? insertions by introducing bulk insert statements

  • double precision problem fixed when user-specified filesize was provided

  • increase number of default pooled threads to 25 so as to avoid gsoap requests backlog

  • transfer multi-hop

  • ATLAS stage-in only feature

  • specify/submit nostreams, timeout, buf size at submission time at the job level

  • activity VO shares

  • fixed delegation issues releated to voms roles

  • introduce bulk cancellations via a file

  • lcgadmin role is now default in fts3config file

  • new client fts-transfer-snapshot to retrieve internal info of FTS3 activity

       fts-transfer-snapshot

         This command returns a snapshot of FTS3 activity per vo and link (number of active

         transfers, queued files, average throughput, etc.).

Usage: fts-transfer-snapshot [options]

Allowed options:

  -h [ --help ]              Print this help text and exit.
  -q [ --quite ]             Quiet operation.
  -v [ --verbose ]         Be more verbose.
  -s [ --service ] arg     Use the transfer service at the specified URL.
  -V [ --version ]          Print the version number and exit.

  --source arg             Specify source site name.
  --destination arg      Specify destination site name.
  --vo arg                   Specify VO name.

  • added time range filter in the monitoring web-app for the optimizer evolution

  • infosys publisher and monitoring using messaging now exist as separate packages and can be installed optionally

Important notes:

  • For sites running on ORACLE

               *scratch the database and re-install the schema from the relevant package

  • For sites running on MySql?

               *If you can drop the schema, drop and re-install

               *If not, then :

1)
drop table t_server_config;
CREATE TABLE t_server_config (retry INTEGER DEFAULT 0,max_time_queue INTEGER DEFAULT 0,global_timeout INTEGER DEFAULT 0,sec_per_mb INTEGER DEFAULT 0);
 
2)
drop table t_hosts;
CREATE TABLE t_hosts (hostname VARCHAR(64) NOT NULL,service_name VARCHAR(64) NOT NULL,beat TIMESTAMP NULL DEFAULT NULL,drain INTEGER DEFAULT 0, CONSTRAINT t_hosts_pk PRIMARY KEY (hostname, service_name));
 
3)
drop table t_activity_share_config;
CREATE TABLE t_activity_share_config (vo VARCHAR(100) NOT NULL PRIMARY KEY,activity_share VARCHAR(255) NOT NULL,active VARCHAR(3) check (active in ('on', 'off')));
 
4)
drop table t_optimize;
CREATE TABLE t_optimize(auto_number INTEGER AUTO_INCREMENT,file_id INTEGER NOT NULL,source_se VARCHAR(150), dest_se VARCHAR(150), nostreams INTEGER DEFAULT NULL,timeout INTEGER DEFAULT NULL, active INTEGER DEFAULT NULL, throughput FLOAT DEFAULT NULL, buffer INTEGER DEFAULT NULL, filesize DOUBLE DEFAULT NULL, datetime TIMESTAMP NULL DEFAULT NULL, udt VARCHAR(3) CHECK (udt in ('on', 'off')), CONSTRAINT t_optimize_pk PRIMARY KEY (auto_number) );
 
5)
ALTER TABLE t_optimize_active MODIFY active INTEGER UNSIGNED DEFAULT 2;
 
6)
This will take long, be patient
ALTER TABLE t_file MODIFY file_metadata VARCHAR(1024);
ALTER TABLE t_file_backup MODIFY file_metadata VARCHAR(1024);
ALTER TABLE t_job MODIFY job_metadata VARCHAR(1024);
ALTER TABLE t_job_backup MODIFY job_metadata VARCHAR(1024);
ALTER TABLE t_file ADD activity VARCHAR(255) DEFAULT "default", ADD COLUMN transferred DOUBLE DEFAULT 0;
ALTER TABLE t_file_backup ADD activity VARCHAR(255) DEFAULT "default", ADD COLUMN transferred DOUBLE DEFAULT 0;
ALTER TABLE t_file DROP INDEX file_tr_host;
ALTER TABLE t_file ADD INDEX t_file_activity(activity), ADD INDEX file_tr_host(transferHost, file_state);
ALTER TABLE t_job ADD INDEX t_job_source_se(source_se),  ADD INDEX t_job_dest_se(dest_se);
 
Packages:
  • fts-client
  • fts-libs
  • fts-devel
  • fts-python
  • fts-server
  • fts-monitoring
  • fts-mysql
  • fts-infosys
  • fts-msg

FTS 3.1.74 released into production

  • reduce send/recv timeout to 110 secs to prevent from killing the clients ungracefully
  • leaking file descriptors when gsoap is set to keep_alive mode
  • monitoring web-app efficiency plots and optimizer decisions 
  • bug-fix for reporting transfer duration using *-status CLI 
  • do not use decimal digits for efficiency calculation 
  • bug-fix for enabling logging in debug mode 
  • prevent a MySql? deadlock and inserting records into retry tables

Important notes:

You will need to manually create the columns below

		ALTER TABLE t_optimize_active ADD datetime TIMESTAMP  NULL DEFAULT NULL;
		ALTER TABLE t_optimize_active ADD message VARCHAR(512);
                ALTER TABLE t_file ADD activity VARCHAR(255) DEFAULT "default";

Pages