CERN Accelerating science

All releases

WebFTS New Release 2.2.0

Task

  • [FTS-141] - Add Data Management operation
  • [FTS-156] - Change the way folders are open

WebFTS New Release 2.1.0

Bug

  • [FTS-154] - Cannot submit jobs in Firefox
  • [FTS-158] - webfts: WebFTS list incorrect meta-data informations for dav listing

Task

  • [FTS-58] - Show which VOMS credentials are initialized for the current proxy
  • [FTS-107] - Integration with CERNBox
  • [FTS-135] - Show Warning when no User certificates are present
  • [FTS-140] - Webfts fails to contact xroot endpoints

        

 

FTS 3.2.30

  • Several bug fixes
  • Query optimizations
  • Bring Online increased reliability and performance
  • Added support for Deletion operations
  • Several improvements in the web monitoring
  • Experimental S3 support
  • REST allows now bulk job polling
    • Using comma-separated job ids (i.e /jobs/job1,job2,job3)

It is recommended to also update to CGSI-gSOAP-1.3.7srm-ifce-1.21.4 and gfal2-2.7.8

FTS 3.2.29

#1240    Bring online timeout is not defaulting to 28800
#1241    Make sure that when multiple files are staged, statuses are not mixed
#1244    replace Queued in overview with Submitted
#1252    Duration: 1413464854 in status
#1261    job status is broken

FTS 3.2.27

-Fts_url_copy Receives SIGSEGV Recursively And Consumes A Lot Of CPU

-Can Now Configure Specific Endpoints To Use IPv6, Not The Whole Service

-Fts-Msg-Cron Has Been Removed From The Package "Fts-Msg", No Longer Needed

-Transfer Log Files  Now Show The Correct IP Of The Hosts (Either IPv4 Or IPv6) Involved In A Transfer

-Bring-Online Job Cancellation Take Into Account That A Single Token May Correspond To More Than One File

-Many Monitoring Web-App New Features

-Different Levels Of Transfer Logs Verbosity Are Now Supported

-Report Properly When VO Extensions Have Expired

-Parallel TCP Streams Optimization, Probing From 1-16 Streams

-Print Stacktrace In The Thread That Generated The Signal

-Allow Fixed Number Of Active Transfer S For Links Only

-Allow The User To Specify "Strict Copy", Meaning Just Copy The File Without Stat, Checksumming, Etc

-Enable Delegation Via REST Interface For FTS3 Cli

-Check For Out Of Disk Space Problems And Report Accordingly

-User Dn Publication Configurable In Messaging, Can Be Disabled Or Enabled On Demand

-Pass To Fts_url_copy The Max Number Of Active Transfers For The Given Link

More Details: Https://Svnweb.Cern.Ch/Trac/Fts3/Milestone/FTS3%203.2.27

 

Important Notes:

1) Mind You Need To Apply First The Database Changes Below (MySql):

SET Storage_engine=INNODB;

alter table t_debug add debug_level  INTEGER DEFAULT 1;

alter table t_optimize add ipv6 VARCHAR(3) CHECK (ipv6 In ('on', 'off'));

alter table t_optimize_active add fixed VARCHAR(3) CHECK (fixed In ('on', 'off'));

alter table t_server_config add show_user_dn VARCHAR(3) CHECK (show_user_dn In ('on', 'off'));

CREATE TABLE t_cloudStorage (
    cloudStorage_name VARCHAR(50) NOT NULL PRIMARY KEY,
    app_key           VARCHAR(255),
    app_secret        VARCHAR(255),
    service_api_url   VARCHAR(1024)
);

CREATE TABLE t_cloudStorageUser (
    user_dn              VARCHAR(700) NULL,
    vo_name              VARCHAR(100) NULL,
    cloudStorage_name    VARCHAR(36) NOT NULL,
    access_token         VARCHAR(255),
    access_token_secret  VARCHAR(255),
    request_token        VARCHAR(255),
    request_token_secret VARCHAR(255),
    FOREIGN KEY (cloudStorage_name) REFERENCES t_cloudStorage(cloudStorage_name),
    PRIMARY KEY (user_dn, vo_name, cloudStorage_name)
);

CREATE TABLE t_optimize_streams (
  source_se    VARCHAR(150) NOT NULL,
  dest_se      VARCHAR(150) NOT NULL,  
  nostreams    INTEGER NOT NULL,   
  datetime     TIMESTAMP  NULL DEFAULT NULL,
  throughput   FLOAT DEFAULT NULL,
  tested       INTEGER DEFAULT 0,
  CONSTRAINT t_optimize_streams_pk PRIMARY KEY (source_se, dest_se, nostreams),
  CONSTRAINT t_optimize_streams_fk FOREIGN KEY (source_se, dest_se) REFERENCES t_optimize_active (source_se, dest_se) ON DELETE CASCADE
);

CREATE INDEX t_optimize_streams_datetime ON t_optimize_streams(datetime);
CREATE INDEX t_optimize_streams_throughput ON t_optimize_streams(throughput);
CREATE INDEX t_optimize_streams_tested ON t_optimize_streams(tested);

2) yum update

fts-*

gfal2-*

srm-Ifce

davix*

CGSI-GSOAP

Packages From Our Staging Repository Http://Fts3-Service.Web.Cern.Ch/Content/Downloads

3.2.26

*Many REST interface optimizations
*Bulk files staging
*Per VO retry global mechanism
*Many web monitoring performance improvements and new features
*Use Exponential Moving Average (EMA) along with weighted-average to avoid frequent "slopes" in the auto-tuning algorithm
*Crash fixed with concurrent delegations
*Log ip of the users in the log file
*Support multi-op and multiple replica jobs using glite/fts3 clients
*Remove weak ciphers from openssl
*fts-transfer-list to include source_se, dest_se in the params list
*Specify max active for a given link, will vary between 2-MAX    
*Many MySql optimizations

 

-Apply the database changes below

SET storage_engine=INNODB;

ALTER TABLE t_optimize_active ADD ema   DOUBLE DEFAULT 0;

ALTER TABLE t_optimizer_evolution ADD agrthroughput   FLOAT DEFAULT NULL;

CREATE INDEX t_optimize_active_datetime  ON t_optimize_active(datetime); 

DROP TABLE t_stage_req;

CREATE TABLE `t_stage_req` (
  `vo_name` varchar(100) NOT NULL,
  `host` varchar(150) NOT NULL,
  `operation` varchar(150) NOT NULL,
  `concurrent_ops` int(11) DEFAULT '0',
  PRIMARY KEY (`vo_name`,`host`,`operation`)
);

CREATE TABLE `t_turl` (

  `source_surl` varchar(150) NOT NULL,
  `destin_surl` varchar(150) NOT NULL,
  `source_turl` varchar(150) NOT NULL,
  `destin_turl` varchar(150) NOT NULL,
  `datetime` timestamp NULL DEFAULT NULL,
  `throughput` float DEFAULT NULL,
  `finish` double DEFAULT '0',
  `fail` double DEFAULT '0',
  PRIMARY KEY (`source_surl`,`destin_surl`,`source_turl`,`destin_turl`),
  KEY `t_url_datetime` (`datetime`),
  KEY `t_url_finish` (`finish`),
  KEY `t_url_fail` (`fail`)
);

ALTER TABLE t_server_config ADD vo_name VARCHAR(100);

DROP TABLE t_dm;

CREATE TABLE `t_dm` (
  `file_id` int(11) NOT NULL AUTO_INCREMENT,
  `job_id` char(36) NOT NULL,
  `file_state` varchar(32) NOT NULL,
  `dmHost` varchar(150) DEFAULT NULL,
  `source_surl` varchar(900) DEFAULT NULL,
  `dest_surl` varchar(900) DEFAULT NULL,
  `source_se` varchar(150) DEFAULT NULL,
  `dest_se` varchar(150) DEFAULT NULL,
  `reason` varchar(2048) DEFAULT NULL,
  `checksum` varchar(100) DEFAULT NULL,
  `finish_time` timestamp NULL DEFAULT NULL,
  `start_time` timestamp NULL DEFAULT NULL,
  `job_finished` timestamp NULL DEFAULT NULL,
  `tx_duration` double DEFAULT NULL,
  `retry` int(11) DEFAULT '0',
  `user_filesize` double DEFAULT NULL,
  `file_metadata` varchar(1024) DEFAULT NULL,
  `activity` varchar(255) DEFAULT 'default',
  `dm_token` varchar(255) DEFAULT NULL,
  `retry_timestamp` timestamp NULL DEFAULT NULL,
  `wait_timestamp` timestamp NULL DEFAULT NULL,
  `wait_timeout` int(11) DEFAULT NULL,
  `hashed_id` int(10) unsigned DEFAULT '0',
  `vo_name` varchar(100) DEFAULT NULL,
  PRIMARY KEY (`file_id`),
  KEY `dm_job_id` (`job_id`),
  KEY `t_dm_all` (`vo_name`,`source_se`,`file_state`),
  KEY `t_dm_source_Se` (`source_se`,`file_state`),
  KEY `t_dm_state` (`file_state`,`hashed_id`),
  CONSTRAINT `t_dm_ibfk_1` FOREIGN KEY (`job_id`) REFERENCES `t_job` (`job_id`)
);

CREATE TABLE t_dm_backup AS (SELECT * FROM t_dm);

-Update all packages below

fts-*

gfal2-*

srm-ifce

CGSI-gSOAP

davix-*

-Restart services

fts-bringonline

fts-msg-bulk

fts-records-cleaner

fts-server

WebFTS New release 2.0.0

Bug

  • [FTS-126] - Refresh Button is not working for Dropbox
  • [FTS-127] - SelectAll does not activate transfers
  • [FTS-128] - Filter->Hide folders does not work for Dropbox
  • [FTS-130] - SE Autocompletion is broken

Improvement

  • [FTS-45] - Keep previous state while browsing between tabs
  • [FTS-86] - keep expanded job details when refreshing 'My Jobs'

New Feature

  • [FTS-37] - Suggest possible VOs to the user when delegating
  • [FTS-92] - Warning for about to expire proxy

Task

  • [FTS-50] - Create the interface for Dropbox-like storages
  • [FTS-89] - Multihop job cannot report info on source_se and dest_se
  • [FTS-90] - WebFTS: keep transfer options in case of resubmission
  • [FTS-112] - Integrate the new FTS REST call to retrieve the latest X ( configurable) jobs
  • [FTS-115] - merge CloudStorage branch
  • [FTS-116] - WebFTS packaging
  • [FTS-125] - Merge site tour, new landing page and all Christina work
  • [FTS-129] - WebFTS configuration file
  • [FTS-131] - Create "official" webfts app for dropbox
  • [FTS-132] - Include News Feed on the landing page

FTS3 new release 3.2.26

 

*Many REST interface optimizations
*Bulk files staging
*Per VO retry global mechanism
*Many web monitoring performance improvements and new features
*Use Exponential Moving Average (EMA) along with weighted-average to avoid frequent "slopes" in the auto-tuning algorithm
*Crash fixed with concurrent delegations
*Log ip of the users in the log file
*Support multi-op and multiple replica jobs using glite/fts3 clients
*Remove weak ciphers from openssl
*fts-transfer-list to include source_se, dest_se in the params list
*Specify max active for a given link, will vary between 2-MAX    
*Many MySql optimizations

 

-Apply the database changes below

SET storage_engine=INNODB;

ALTER TABLE t_optimize_active ADD ema   DOUBLE DEFAULT 0;

ALTER TABLE t_optimizer_evolution ADD agrthroughput   FLOAT DEFAULT NULL;

CREATE INDEX t_optimize_active_datetime  ON t_optimize_active(datetime); 

DROP TABLE t_stage_req;

CREATE TABLE `t_stage_req` (
  `vo_name` varchar(100) NOT NULL,
  `host` varchar(150) NOT NULL,
  `operation` varchar(150) NOT NULL,
  `concurrent_ops` int(11) DEFAULT '0',
  PRIMARY KEY (`vo_name`,`host`,`operation`)
);

CREATE TABLE `t_turl` (

  `source_surl` varchar(150) NOT NULL,
  `destin_surl` varchar(150) NOT NULL,
  `source_turl` varchar(150) NOT NULL,
  `destin_turl` varchar(150) NOT NULL,
  `datetime` timestamp NULL DEFAULT NULL,
  `throughput` float DEFAULT NULL,
  `finish` double DEFAULT '0',
  `fail` double DEFAULT '0',
  PRIMARY KEY (`source_surl`,`destin_surl`,`source_turl`,`destin_turl`),
  KEY `t_url_datetime` (`datetime`),
  KEY `t_url_finish` (`finish`),
  KEY `t_url_fail` (`fail`)
);

ALTER TABLE t_server_config ADD vo_name VARCHAR(100);

DROP TABLE t_dm;

CREATE TABLE `t_dm` (
  `file_id` int(11) NOT NULL AUTO_INCREMENT,
  `job_id` char(36) NOT NULL,
  `file_state` varchar(32) NOT NULL,
  `dmHost` varchar(150) DEFAULT NULL,
  `source_surl` varchar(900) DEFAULT NULL,
  `dest_surl` varchar(900) DEFAULT NULL,
  `source_se` varchar(150) DEFAULT NULL,
  `dest_se` varchar(150) DEFAULT NULL,
  `reason` varchar(2048) DEFAULT NULL,
  `checksum` varchar(100) DEFAULT NULL,
  `finish_time` timestamp NULL DEFAULT NULL,
  `start_time` timestamp NULL DEFAULT NULL,
  `job_finished` timestamp NULL DEFAULT NULL,
  `tx_duration` double DEFAULT NULL,
  `retry` int(11) DEFAULT '0',
  `user_filesize` double DEFAULT NULL,
  `file_metadata` varchar(1024) DEFAULT NULL,
  `activity` varchar(255) DEFAULT 'default',
  `dm_token` varchar(255) DEFAULT NULL,
  `retry_timestamp` timestamp NULL DEFAULT NULL,
  `wait_timestamp` timestamp NULL DEFAULT NULL,
  `wait_timeout` int(11) DEFAULT NULL,
  `hashed_id` int(10) unsigned DEFAULT '0',
  `vo_name` varchar(100) DEFAULT NULL,
  PRIMARY KEY (`file_id`),
  KEY `dm_job_id` (`job_id`),
  KEY `t_dm_all` (`vo_name`,`source_se`,`file_state`),
  KEY `t_dm_source_Se` (`source_se`,`file_state`),
  KEY `t_dm_state` (`file_state`,`hashed_id`),
  CONSTRAINT `t_dm_ibfk_1` FOREIGN KEY (`job_id`) REFERENCES `t_job` (`job_id`)
);

CREATE TABLE t_dm_backup AS (SELECT * FROM t_dm);

-Update all packages below

fts-*

gfal2-*

srm-ifce

CGSI-gSOAP

davix-*

-Restart services

fts-bringonline

fts-msg-bulk

fts-records-cleaner

fts-server

 

FTS3 REST 3.2.3 tagged

Bug

  • [FTS-30] - Validate that the proxy is properly signed
  • [FTS-35] - When listing a davs:// endpoint, delegation stop working
  • [FTS-60] - WebDAV third party copy delegation is broken
  • [FTS-67] - REST CLI does not send properly formed json metadata

Improvement

  • [FTS-27] - Answer with 419 Authentication Timeout when a new delegation is needed
  • [FTS-31] - Use random distribution for hashed_id
  • [FTS-33] - In submission, insert OptimizerActive only for unique (source_se, dest_se)
  • [FTS-34] - In submission, use bulk inserts
  • [FTS-36] - Delegation id must be generated using all fqans
  • [FTS-66] - REST CLI doesn't work in SL5
  • [FTS-76] - Improve rest client error handling

New Feature

  • [FTS-24] - VOMS proxy support
  • [FTS-69] - REST is missing multihop submissions

Task

  • [FTS-26] - Improve logging for REST
  • [FTS-29] - selinux rpm for test must let apache run voms-proxy-init and create the proxy
  • [FTS-32] - Check queries being generated by SqlAlchemy
  • [FTS-75] - REST: Provide reference Python functions for LHCb

User documentation

  • [FTS-25] - Improve REST API documentation

FTS3 announcement of updates

FTS3 updates will be announced in:

 

Pages