The Implementation of Quality Assessment Analysis and Monitoring System for Realistic Reproduction Effects


Aria Seo1, Yei-Chang Kim2*

1Ph.D Course, Dept. of Techno-Management Cooperative Course, Dongguk University123 Dongdae-ro, Gyeongju-si, Gyeongsangbuk-do 38066, Korea

2Professor, Dept. of Management, Dongguk University 123 Dongdae-ro, Gyeongju-si,

Gyeongsangbuk-do 38066, Korea

*Corresponding Author E-mail:,



Background/Objectives: In this paper, an evaluation system for quantitative measurement of realistic reproduction effects is constructed, and a system that can be applied is presented. Methods/Statistical analysis: When the degree of feeling (wind, vibration, motion, water, flash, etc.) according to the experience of realistic media is transmitted through the user's mobile device, the operating system collects the feeling index of the reality reproduction effect. Then, the collected feeling index was analyzed by database. We implemented a system that reflects the result to the system and automatically corrects the metadata according to the user's request. Findings: Through the implementation of the system, we were able to grasp the satisfaction of users who will experience real media. It also enables quantitative quality assessment of the realistic reproduction effect that was not even existing standard. Improvements/Applications: It is possible to produce a realistic media that provides a sensory effect suited to the user's needs through accumulated database of user experience.


KEYWORDS: Realistic media, Realistic reproduction effect, RREA, SEM.




Realistic media can enhance user's sense of reality and immersion by projecting various effects such as multi-channel audio, wind, water, vibration, and flash in addition to high-quality video in media contents1,2,3. At present, the reproducibility effect of the realistic media contents is produced according to the subjective aspect of the developer without considering the satisfaction degree of the user's sensation, and the quality of the sensory effect is not appropriate4.


In order to apply the actual user's degree of feeling, statistical analysis should precede quantitative measurement of opinions of users. Then, it is necessary to modify the realistic metadata according to the result value5,6. However, there are few cases in which realistic media content producers measured user satisfaction. It cannot be implemented due to the time and cost that occurs in updating user’s sense of satisfaction in accordance with the needs of the user7,8.


In this paper, we have developed a system for analyzing user's feeling satisfaction and controlling realistic metadata based on this. Based on various statistical data obtained from the monitoring system for managers, realistic media contents with user’s high feeling satisfaction can be produced. In other words, the feeling index of the reality reproduction effect that the user feels through the mobile device is collected, converted into a database, and the analysis result is reflected in the system in real time, so that the metadata can be automatically corrected.





2.1. Model of quality assessment analysis system:

The RREA(Realistic reproduction Effect Assessment) platform proposed in this paper can specify evaluation indexes for realistic media producers to enhance the realistic reproduction effect for users' satisfaction evaluation(see Figure 1). The user’s evaluation index can modify the index based on the assessment result of a large number of users. Therefore, as the evaluation of various users progresses, the feeling quality can be improved9,10,11.



Figure 1.RREA Platform Structure


The Web-based Assessment Module consists of a Survey Node and an Assessment Node. The Survey Node is a node that generates an evaluation index for evaluating the realistic reproduction effect for the user. Various evaluation indexes are needed to evaluate realistic media of various genres. Therefore, the system manager can generate the evaluation index template classified according to the genre according to the needs of the realistic media producer. The Assessment User connects to the Survey Node using the mobile device and performs the evaluation. Data that has been evaluated in the Survey Node is sent to the Assessment Node.


The Assessment Node creates a virtual machine in the same form as the virtual machine created in the Survey Node. The virtual machine performs the function of quantifying and storing evaluation indexes. The virtual machines created in the Assessment Node are quantified and stored according to the producer's request. Therefore, the created virtual machine keeps synchronized with the virtual machine created in the Survey Node, but it can always be switched without maintaining the same type of virtual machine.


The user's evaluation data is collected through the Web-based Assessment Module. The evaluation data for the realization reproduction effect is performed the statistical analysis through the realistic reproduction effect metadata module. The evaluation result is stored in the Media Database of the CMS Node.


Based on the evaluation data, Analyzer Node analyzes the effect of realistic reproduction effect on user’s satisfaction. It is divided into positive and negative factors, and analyzes the sensory quality for user's feeling. The analyzer node's virtual machine synchronizes with the virtual machine of the web-based evaluation module and analyzes the evaluation result through the management node. Through this process, the producer can monitor the evaluation result in real time and can modify it to improve the satisfaction degree according to the evaluation result.


CMS Node consists of Media Database, Media Storage and Stream System. It can actively add and delete according to the situation. The Media Database stores the metadata of the realistic reproduction effect and inputs and manages the modified reality reproduction metadata according to the analysis result of the Analyzer Node. Since real-life media are large-volume contents, it is difficult to store and manage the original type media. Therefore, media storage encodes realistic media. The Stream System is a transmission system that enables the assessment user to watch realistic media images stored in Media Storage in real time using mobile devices.


The system manager uses the Management Node to manage and control all the Nodes. All nodes create and manage virtual machines. In addition, the RREA platform module controls Web-UI to add a new node. The producer who wants to access the management node can access the Web-UI by granting the system manager an extra privilege. Therefore, the producer can generate a template suitable for performing a sensory quality evaluation, and can also use the template generated by another producer to evaluate the sensory quality of the realistic reproduction effect.



Figure 2.Scenario for Evaluating the Quality of Realistic Reproduction Effect



Figure 3.Quality Assessment Analyses Algorithm

In order to utilize the quality assessment system for measuring the realistic reproduction effect, information is obtained using the scenario as shown in Figure 2. According to the scenario, the manager can receive various statistical analysis results of the customers and users based on the evaluation information. In addition, the manager can confirm an appropriate reality reproduction effect by the realistic media genre, and can introduce it from the content production stage.


2.2. Assessment quality analysis Algorithm:

Figure 3 presents the assessment quality analysis algorithm for analyzing the user evaluation index, and analyzes the input evaluation data12.


The evaluation data processing algorithm that processes the results obtained from the quality analysis presented in Figure 4 takes steps to improve degree of feeling satisfaction.





Figure 4.Assessment Data Processing Algorithm


2.3. Monitoring system model for the realistic reproduction effect:

User evaluation is performed as a method for evaluating the realistic reproduction effect. Figure 5 shows the system architecture developed in this paper. The evaluator inputs the age, the existing experience frequency, the genre of the content to be experienced to his mobile devices before experiencing the realistic media13. The evaluation page provided is web-based and this assessment module consists of Survey Node and Assessment Node. The Survey Node is a node that generates each evaluation index to evaluate the realistic reproduction effect to users. Different types of evaluation indexes are needed to evaluate realistic media with different genres14. Therefore, the system manager can generate the evaluation index classified based on the genre.


After experiencing realistic media, the user connects to the Survey Node and performs evaluation. The data that has been evaluated is processed by the Assessment Node. The Assessment Node is quantified and stored as requested by the producer. Therefore, it maintains synchronization with Survey Node, but it can be actively switched. The evaluation result of the user is collected through the web - based evaluation module, and the evaluation result of the reality reproduction effect is analyzed through the metadata improvement module. Finally, the completed result is stored in the Media Database of the CMS Node.


Figure 5.The System Structure Presented in This Paper


The Analyzer Node analyzes the feel of the realistic reproduction effect on the user according to the evaluation result. It is divided into positive factor and negative factor, and the user's perception level is quantified and intuitively displayed.


The CMS Node consists of Media Database and Media Storage, and can be actively added and deleted depending on the situation.


2.4. Monitoring System Algorithm for Evaluating Reproduction Effects:

The system developed in this paper analyzes the evaluation data of the realistic reproduction effect of specific realistic media using the SEM (Sensory Effect Metadata) Analyzer when the measurement data of the level set by the system manager is collected15. As a result of the analysis, we need to modify the SEM through SEvino when it is necessary to adjust the realistic reproduction effect. The modified metadata is transferred to the SEM DB and stored. The continuously collected evaluation DB will modify the metadata of the SEM DB. However, if it does not exceed the threshold value range, do not modify it. According to the evaluation result on the realistic reproduction effect, it improves the service quality by modifying the attribute value of the metadata for the realistic reproduction effect.



Figure 6.Metadata of Realistic Reproduction Effect in SEvino

The SEM Analyzer adjusts the attribute values of the metadata based on the result of the realistic reproduction effect evaluated by the user. As shown in Figure 6, the metadata attribute value of the realistic reproduction effect written in XML is composed of effect type, operation status, strength, range, start point, and end point. Modify the attribute value according to the user's evaluation. If there are many responses that the wind strength is strong in the user quality evaluation, change the intensity of the wind by modifying the intensity-value. Also, if there are many responses that the feeling of immersion is low due to the wind effect, it is possible to remove the effect by changing the wind effect to false. As a result, it is possible to improve the quality of the user's feeling by changing the designated property value according to the response of the user evaluation.



3.1. Extract evaluation element of realistic media contents:

In order to evaluate realistic media, various kinds of contents genre and kinds of reality reproduction devices should be defined16. In this paper, realistic media contents are classified into six genres such as action, adventure, comedy, thriller, fantasy, and science fiction. Also, the realistic reproduction devices are divided into 8 types such as Wind, Bubbles, Lightning, Fog Scent, Air Shot, Water Shot, and Moving. According to the result of quality evaluation for realistic reproduction effect, it is necessary to designate evaluation factor to reapply the effect or introduce it into the content production process17. Therefore, it included objective information such as general information such as age and gender of the evaluator and external factors.


In this paper, we evaluated six items such as immersion, usefulness, strength, duration, sink, and pleasure for quality evaluation of realistic reproduction effect.


In the assessment screen of the system developed in this paper, as shown in Table 1, a quality assessment matrix was applied to evaluate the realistic reproduction effect. Through the matrix, evaluation of functional requirements and quality requirements can be integrated to improve the reliability and usability of evaluation. Table 2 is a screen for evaluating the satisfaction of each item on a 5-point scale.


Table 1: Quality Assessment Matrix of Realistic Effect of Simulation

Matrix Sectin

Realistic effect of simulation








Air shot

Water shot











Table 2:Overall Satisfactory of the Quality of Each Function

Assessment field

Assessment Score

Target Score

Achived (%)

Overall satisfactory level








































3.2. Implementation of the application for the evaluation analysis system:

In this paper, we use smart mobile device of user to measure evaluation items. The page for the survey was developed as a web-based application to enable various mobile devices. Figure 7 is the developed mobile application screen.




4. Implementation of Quality Assessment Monitoring System:

4.1. Realistic reproduction effect tool and monitoring system:

In this paper, we implemented the system using open source SEvino. You can see the Authoring Tool in Figure 8. In Figure 8(A), the metadata applied to realistic media contents can be confirmed. The metadata for the realistic media effect can be confirmed through Figure 8 (B).

(A) Web Page Composition

(B) Screen for quality assessment

(C) Result screen for quality assessment

Figure 7.Mobile Web Page Screen


(A) Initial set realistic media metadata

(B) Applying metadata of modified reality media

Figure 8.Authoring Tool for Realistic Reproduction Effect


4.1.System simulation:

In this paper, we evaluate the haptic effect of the realistic reproduction effect in the realistic experience hall. The realistic experience hall was equipped with 4D screen, 5.1 channel audio, and equipment to reproduce Wind, Vibration, Light, Temperature, Water Sprayer, Scent, Fog effect. For the simulation, 118 subjects (66 men and 52 women) were evaluated. The realistic contents consisted of 15 minutes, and initial values were set in consideration of Wind, Vibration, Light, Temperature, Water Sprayer, Scent, and Fog effect.



In (A) of Figure 9, the evaluator inputs basic information such as sex, age and genre. (B) is a screen where the evaluator experiences the real media. (C) is a screen that allows evaluation to be performed on immersion, usability, strength, duration, and synchronization.


Assessment results can be viewed in real time on a computer or mobile device. As shown in (A) of Figure 10., the numbers of evaluators, age, and genres and the evaluation values of each realistic reproduction effect can be confirmed. In Figure (B), it is possible to exclude inadequate respondents because they can see the evaluation data evaluated by the evaluators as a whole.

(A) User survey input screen

(B) Watching and experiencing realistic media

(C) Assessment of the realistic Reproduction effect

Figure 9.Entering Assessment Items and System Test


(A) Evaluation result of realistic reproduction effect

(B) Checking evaluation data by evaluator

Figure 10.Monitoring Screen Showing Evaluation Result


(A) SEM value before evaluation

(B) SEM value after evaluation

Figure 11.SEM Setting Values Before and After Evaluation


Figure 11(A) is a screen showing the degree of satisfaction with the realistic reproduction effect according to the initially set SEM value. Figure 11(B) is a screen showing the satisfaction of the realistic reproduction effect after modifying the SEM value according to the evaluator's request. It can be understood that the satisfaction level of the users is increased.



In this paper, we have developed a system to evaluate and apply QoE based assessment system for quantitative measurement of realistic reproduction effect. As a result, the frequency of experience of users experiencing realistic media increases, and the degree of the user's perception can be accurately grasped. In addition, it has made it possible to evaluate the quality of the realistic reproduction effect that was not even exists in the past. As a result, it can be expected that the quality of realistic media can be improved because it considers not only the visual and auditory sense but also the sensory effect during the production of realistic media.


In the future, the reliability of the assessment system should be improved by performing various evaluations based on the developed assessment criteria. Also, it is necessary to evaluate the performance of the realistic media authoring tool applying modified metadata.



1.   Aria Seo, Yei-Chang Kim, A Study on the Contextual Information Sharing System for the Provide Matched Information through Contextual Data from Multi-Users, International Journal of Pharma and Bio Sciences (IJPBS), 2017, SP05/Apr/2017, pp.480-484.

2.   Markus Waltl, Christian Timmerer, Benjamin Rainer, Hermann Hellwagner, Sensory effects for ambient experiences in the World Wide Web, 2014, Multimedia tools and applications, Vol. 70, No. 2, pp.1141-1160.

3.   Markus Waltl, Christian Timmerer, Benjamin Rainer, Hermann Hellwagner, SENSORY EFFECT DATASET AND TEST SETUPS, QoMex International Workshop on Quality of Multimedia Experience, 2012, pp.115-120.

4.   Markus Waltl, Christian Timmerer, Hermann Hellwagner, A test-bed for quality of multimedia experience evaluation of Sensory Effects, 2009,QoMex International Workshop on Quality of Multimedia Experience, pp.145-150.

5.   Markus Waltl, Benjamin Rainer, Christian Timmerer, Hermann Hellwagner, An end-to-end tool chain for Sensory Experience based on MPEG-V, Signal Processing: Image Communication, 2013, Vol. 8, No. 2, pp.136-150.

6.   SeokmoGu, Yei-Chang Kim, A Study on the Quality assessment Improvement Scheme of Realistic reproduction Effects based on User Quality Assessment, Indian Journal of Science and Technology, Indian Society of Education and Environment, 2015, Vol. 8, No. 6, pp.1-7.

7.   SeokmoGu, Yei-Chang Kim, A Study on Transmission System for Realistic Media Effect reproduction, Indian Journal of Science and Technology, Indian Society of Education and Environment, 2015, Vol. 8, No. 5, pp.28-32.

8.   Aria Seo, Yei-Chang Kim, The Implementation of Assessment System based on QoE for the Quantitative Measurement of Realistic Media Effects, e-Business Study, 2016, Vol.17, No.2,pp.37-55.

9.   Akira Takahashi, David Hands, Vincent Barriac, Standardization Activities in the ITU for a QoE Assessment of IPTV, IEEE Communications Magazine, 2008, pp.78-84.

10.  Andrew Davis, Damien Bayart, David Hands, Quality Assurance for IPTV, IEEE. Broadband Multimedia Systems and Broadcasting, 2009, pp.1-11.

11.  ITU-T Recommendation, Subjective Video Quality Assessment Methods for Multimedia Applications, 2008, p.910.

12.  Sang-yong Ha, Chin Chol Kim, Dong-Jin Shin, Yong-hyun Jo, Byeong-heeRoh, Video QoE Measurement Algorithm by Parameter Matching for IPTV Services, Korea Institute Of Communication Sciences, The Journal of The Korean Institute of Communication Sciences, 2011, Vol. 36, No. 5, pp.451-463.

13.  Aria Seo, Yei-Chang Kim, The System Structure for Processing of Varying Context from Multi Users and Sensors in IoT Environment, Asia-pacific Proceedings of Applied Science and Engineering for Better Human Life, 2016, Vol. 10, pp138-142.

14.  Shinyoung Chung, JuhyunEune, Framework for the Quantitative Evaluation of Media Arts, Korean Society of Design Science, Archives of Design Research, 2006, Vol. 19, No. 2, pp.139-150.

15.  ITU-R Recommendation, Technical characteristics for an automatic identification system using time-division multiple access in the VHF maritime mobile band, International Telecommunication Union(ITU), 2010, pp.1371-1374.

16.  SajadKhorsandroo, Rafidah Md Noor, SayidKhorsandroo, A Generic Quantitative Relationship to Assess Interdependency of QoE and QoS, KSII Transactions on Internet and Information Systems (TIIS), 2013, Vol. 7 No. 2, pp.327-346.

17.  ISO/IEC, Information Technology –Media Context and Control –Part 3: Sensory Information, 2011, pp.23005-30.










Received on 18.07.2018         Modified on 05.09.2018

Accepted on 09.10.2018      © RJPT All right reserved

Research J. Pharm. and Tech. 2019; 12(3): 1066-1074.

DOI: 10.5958/0974-360X.2019.00175.6