From Vision to Content: Construction of Domain-Specific Multi-Modal Knowledge Graph
Knowledge graphs are usually constructed to describe the various concepts that exist in real world as well as the relationships between them. There are many knowledge graphs in specific fields, but they usually pay more attention on text or structured data, ignoring the image vision information, and...
Main Authors: | , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
IEEE
2019-01-01
|
Series: | IEEE Access |
Subjects: | |
Online Access: | https://ieeexplore.ieee.org/document/8788525/ |
id |
doaj-8d66c572f11b4c9985ee92d329b5e309 |
---|---|
record_format |
Article |
spelling |
doaj-8d66c572f11b4c9985ee92d329b5e3092021-04-05T17:04:31ZengIEEEIEEE Access2169-35362019-01-01710827810829410.1109/ACCESS.2019.29333708788525From Vision to Content: Construction of Domain-Specific Multi-Modal Knowledge GraphXiaoming Zhang0Xiaoling Sun1https://orcid.org/0000-0001-6220-8382Chunjie Xie2Bing Lun3School of Information Science and Engineering, Hebei University of Science and Technology, Shijiazhuang, ChinaSchool of Information Science and Engineering, Hebei University of Science and Technology, Shijiazhuang, ChinaSchool of Information Science and Engineering, Hebei University of Science and Technology, Shijiazhuang, ChinaSchool of Information Science and Engineering, Hebei University of Science and Technology, Shijiazhuang, ChinaKnowledge graphs are usually constructed to describe the various concepts that exist in real world as well as the relationships between them. There are many knowledge graphs in specific fields, but they usually pay more attention on text or structured data, ignoring the image vision information, and cannot play an adequate role in the emerging visualization applications. Aiming at this issue, we design a method that integrates image vision information and text information derived from Wikimedia Commons to construct a domain-specific multi-modal knowledge graph, taking the metallic materials domain as an example to illustrate the method. The text description of each image is regarded as its context semantic to acquire the image's context semantic labels based on the DBpedia resource. Furthermore, we adopt deep neural network model instead of simple visual descriptors to acquire the image's visual semantic labels using the concepts from WordNet. In order to fuse the visual semantic labels and context semantic labels, a path-based concept extension and fusion strategy is proposed based on the conceptual hierarchies of WordNet and DBpedia to obtain the effective extension concepts as well as the links between them, increasing the scale of the knowledge graph and enhancing the correlation between images. The experimental results show that the maximum extension level has a significant impact on the quality of the generated domain knowledge graph, and the best extension level number is respectively determined for both DBpedia and WordNet. In addition, the results of this paper are compared with IMGpedia to further show the effectiveness of the proposed method.https://ieeexplore.ieee.org/document/8788525/Knowledge graphinformation fusionmulti-modal knowledgemetallic materials knowledge |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Xiaoming Zhang Xiaoling Sun Chunjie Xie Bing Lun |
spellingShingle |
Xiaoming Zhang Xiaoling Sun Chunjie Xie Bing Lun From Vision to Content: Construction of Domain-Specific Multi-Modal Knowledge Graph IEEE Access Knowledge graph information fusion multi-modal knowledge metallic materials knowledge |
author_facet |
Xiaoming Zhang Xiaoling Sun Chunjie Xie Bing Lun |
author_sort |
Xiaoming Zhang |
title |
From Vision to Content: Construction of Domain-Specific Multi-Modal Knowledge Graph |
title_short |
From Vision to Content: Construction of Domain-Specific Multi-Modal Knowledge Graph |
title_full |
From Vision to Content: Construction of Domain-Specific Multi-Modal Knowledge Graph |
title_fullStr |
From Vision to Content: Construction of Domain-Specific Multi-Modal Knowledge Graph |
title_full_unstemmed |
From Vision to Content: Construction of Domain-Specific Multi-Modal Knowledge Graph |
title_sort |
from vision to content: construction of domain-specific multi-modal knowledge graph |
publisher |
IEEE |
series |
IEEE Access |
issn |
2169-3536 |
publishDate |
2019-01-01 |
description |
Knowledge graphs are usually constructed to describe the various concepts that exist in real world as well as the relationships between them. There are many knowledge graphs in specific fields, but they usually pay more attention on text or structured data, ignoring the image vision information, and cannot play an adequate role in the emerging visualization applications. Aiming at this issue, we design a method that integrates image vision information and text information derived from Wikimedia Commons to construct a domain-specific multi-modal knowledge graph, taking the metallic materials domain as an example to illustrate the method. The text description of each image is regarded as its context semantic to acquire the image's context semantic labels based on the DBpedia resource. Furthermore, we adopt deep neural network model instead of simple visual descriptors to acquire the image's visual semantic labels using the concepts from WordNet. In order to fuse the visual semantic labels and context semantic labels, a path-based concept extension and fusion strategy is proposed based on the conceptual hierarchies of WordNet and DBpedia to obtain the effective extension concepts as well as the links between them, increasing the scale of the knowledge graph and enhancing the correlation between images. The experimental results show that the maximum extension level has a significant impact on the quality of the generated domain knowledge graph, and the best extension level number is respectively determined for both DBpedia and WordNet. In addition, the results of this paper are compared with IMGpedia to further show the effectiveness of the proposed method. |
topic |
Knowledge graph information fusion multi-modal knowledge metallic materials knowledge |
url |
https://ieeexplore.ieee.org/document/8788525/ |
work_keys_str_mv |
AT xiaomingzhang fromvisiontocontentconstructionofdomainspecificmultimodalknowledgegraph AT xiaolingsun fromvisiontocontentconstructionofdomainspecificmultimodalknowledgegraph AT chunjiexie fromvisiontocontentconstructionofdomainspecificmultimodalknowledgegraph AT binglun fromvisiontocontentconstructionofdomainspecificmultimodalknowledgegraph |
_version_ |
1721540381312548864 |