Nov 2018 Data has been computed using the sequences from the ITS4S database available on https://www.cdvl.org/ : use the keyword "its4", there are 8 files containing uncompressed videos. Note that the files are in 7zip format (not plain zip format, as the .zip extension would suggest) For the experimental settings used to create the sequences, the naming conventions, etc., please refer to M. Pinson, "ITS4S: A Video Quality Dataset with Four-Second Unrepeated Scenes", TM-18-532, 2018 For the case of VMAF, all data have been computed using the all-in-one version of the software (vmafossexec), producing results in the .json format. Temporal pooling has been computed by arithmetic average of the values over the whole set of frames (e.g. for vmaf, ci95_low, ci95_high, etc.) Original content was 720p 24fps. All PVSs, after scaling/compression, have already been upscaled to the original size (720p). Thus no processing had to be done in this regard. NB: the reference has to be temporally aligned to the PVS. It is systematically mis-aligned by 1 frame. The xlsx file with the subjective scores is available on https://www.cdvl.org/ in the "ITS4S - Video Quality Dataset - compressed" set. Note that the csv file has many fields with the same value, since for simplicity I run ./vmafossexec with --psnr --ssim --ms-ssim using the 3 different VMAF models 0.6.0, 0.6.1, 0.6.2, and I extracted all the information together. Of course, psnrV062 = psnrV061 = psnrV060; ssimV062 = ssimV061 = ssimV060; ms_ssimV062 = ms_ssimV061 = ms_ssimV060. Also, psnrV062 = psnr, but this is not the case for ssim and msssim as computed by vqmt. For the MOS values, in the VQEG Nov 2018 presentation I always used myMOS_NTIA, i.e., only the objective scores computed in the NTIA lab. The csv file contains the following information: 1 name : the identification of the sequence - meaning explained in the NTIA report 2 psnrV062 : field "psnr" in the json file 3 ssimV062 : field "ssim" in the json file 4 ms_ssimV062 : field "ms_ssim" in the json file 5 stddevV062 : field "stddev" in the json file 6 vmafV062 : field "vmaf" in the json file (using model vmaf_model-1.3.10/vmaf_rb_v0.6.2/vmaf_rb_v0.6.2.pkl ) 7 ci95_lowV062 : field "ci95_low" in the json file 8 ci95_highV062 : field "ci95_high" in the json file 9 baggingV062 : field "bagging" in the json file VMAF, derived from the bootstrap aggregation as computed by the vmaf software using model vmaf_model-1.3.10/vmaf_rb_v0.6.2/vmaf_rb_v0.6.2.pkl 10 psnrV061 : field "psnr" in the json file 11 ssimV061 : field "ssim" in the json file 12 ms_ssimV061 : field "ms_ssim" in the json file 13 vmafV061 : field "vmaf" in the json file (using model vmaf_model-1.3.10/vmaf_v0.6.1.pkl ) 14 psnrV060 : field "psnr" in the json file 15 ssimV060 : field "ssim" in the json file 16 ms_ssimV060 : field "ms_ssim" in the json file 17 vmafV060 : field "vmaf" in the json file (using model vmaf_model-1.3.10/vmaf_v0.6.0.pkl ) 18 psnr : PSNR as computed by the vqmt tool (arithmetic average over all frames) 19 ssim : SSIM as computed by the vqmt tool (arithmetic average over all frames) 20 vifp : VIF as computed by the vqmt tool (arithmetic average over all frames) 21 msssim : MS-SSIM as computed by the vqmt tool (arithmetic average over all frames) 22 pvqm : PVQM as computed by this tool: http://media.polito.it/wordpress/wp-content/uploads/2013/12/pvqm_EM_v1.3.zip 23 vqm100fr : NTIA VQM computed by the binary executable vqm (assuming 25 fps, and duplicating some frames at beginning and end to reach the minimum length of 100 frames) 24 ci95_sizeV062 : ci95_highV062-ci95_lowV062 25 MOSabs : MOS value provided by the .xlsx file accompanying the ITS4S database 26 SOSabs : SOS value (standard deviation of opinion scores) provided by the .xlsx file accompanying the ITS4S database 27 Nsubj : total number of subjective scores present in the .xlsx file for that particular name 28 bitrate : the encoding bitrate (derived by the name) 29 NsubjNTIA : total number of subjective scores created by NTIA (subject ID < 100) 30 myMOS_NTIA : average of subjective scores for all subjects with ID <100 31 mySOS_NTIA : stddev of subjective scores for all subjects with ID <100 32 myCI95min_NTIA : min of 95% confidence interval for subjects with ID <100: myCI95min_NTIA = myMOS_NTIA - 1.96*mySOS_NTIA/sqrt(Nsubj) 33 myCI95max_NTIA : max of 95% confidence interval for subjects with ID <100: myCI95min_NTIA = myMOS_NTIA + 1.96*mySOS_NTIA/sqrt(Nsubj) 34 myCI95size_NTIA : myCI95max_NTIA-myCI95min_NTIA 35 NsubjAGH : total number of subjective scores created by AGH (subject ID > 100) 36 myMOS_AGH : average of subjective scores for all subjects with ID >100 37 mySOS_AGH : stddev of subjective scores for all subjects with ID >100 38 myCI95min_AGH : min of 95% confidence interval for subjects with ID <100: myCI95min_AGH = myMOS_AGH - 1.96*mySOS_AGH/sqrt(NsubjAGH) 39 myCI95max_AGH : max of 95% confidence interval for subjects with ID <100: myCI95min_AGH = myMOS_AGH + 1.96*mySOS_AGH/sqrt(NsubjAGH) 40 myCI95size_AGH : myCI95max_AGH-myCI95min_AGH 41 session : a single letter among B,C,E,M,N,O,P,S,T (Broadcast, Chance, Everglades etc...): derived from the name 42 srcID : id of the source (2 digits, following the "src" string in the name) 43 frnum : number of frames available in the encoded sequence (PVS) 44 effbitrate : actual bitrate of the PVS (.mp4 file size in bits / frnum * fps ) (fps=24)