一种基于Lifelogging视频的文本标签生成模型

A Text Label Generation Model Based on Lifelogging Videos

摘要: 从2011年开始，我们发起了一个收集个人信息生活记录数据的项目，该项目收集了22位志愿者的4万条lifelogging数据。随着时间的推移，志愿者的lifelogging数据越来越多，其中收集到的视频就多达3020条，想要搜索这些lifelogging数据中的视频变得非常困难。因此，我们提出了一种视频分解 + 图像分析的模型Liu-VTM (Video Tags Model)，该模型从lifelogging视频中筛选能够代表该视频内容的关键帧，并依据关键帧进行图像识别得到视频的标签，最后可以通过标签直接检索到相应的视频。在本次实验中我们探讨了多种视频选取关键帧的方法对模型的影响，并提出了一个新的评价指标“最佳内容覆盖率”用于评价lifelog领域内视频选取到的关键帧的性能。我们的实验结果证明了Liu-VTM模型可以有效对lifelogging数据集打上视频标签并依据标签直接检索到相应视频。

Abstract: Since 2011, we have initiated a project to collect personal lifelogging data, gathering 40,000 lifelogging entries from 22 volunteers. Over time, the amount of lifelogging data from the volunteers has increased, including as many as 3020 videos, making it extremely difficult to search through these lifelogging videos. Therefore, we propose a video decomposition and image analysis model called Liu-VTM (Video Tags Model). This model selects keyframes from lifelogging videos that represent the content of the video and uses image recognition on these keyframes to generate video tags. These tags can then be used to directly retrieve the corresponding videos. In this experiment, we explored various methods for selecting keyframes from videos and proposed a new evaluation metric called “Optimal Content Coverage Rate” to assess the performance of keyframe selection in the lifelogging domain. Our experimental results demonstrate that the Liu-VTM model can effectively tag videos in lifelogging datasets and retrieve the corresponding videos based on these tags.