A machine learning model has been developed for the analysis of composed ornaments [Khan M. S., Emonet R. et Fournel T., 2021] in order to help the human experts during the process of assignment to the publisher, Marc-Michel Rey [Bahier-Porte et Vial-Bonacci, 2019]. The model is designed for vignette pattern detection with bounding box and probability assignment,then reconstruction from features when the assigned probability is greater than a predefined threshold value. Such a decomposition into vignettes can be achieved after learning the parameters of the model from a dictionary of vignettes used by M.-M. Rey. Out-of-dictionary or suspicious vignettes are so missing or badly reproduced in the reconstructed image (Fig. 1).
Figure 1. (Left) An input image composed with vignettes randomly distributed, with the detected bounding boxes in red, out-of-dictionary vignettes (ground truth) being indicated by bounding boxes in blue. (Right) The reconstructed image.
The architecture of the model is an object detector placed upstream of a decoder to strengthen both detection and copy-and-past of vignettes in the dictionary. More precisely, a Single Shot Multibox Detector [Liu et al 2016] is used to predict boxes bounding predicted vignettes in the dictionary. In the detection part, some layers are added to align the predicted bounding boxes [He et al 2017] and transform the feature maps into 1 × 1 × 1024 feature vectors. In the decoding part, the feature vectors are fed into three fully connected linear layers then reshaped into 128×128 reconstructed vignettes. The Detector-Encoder was first trained separately by using Distance IoU [Zheng et al 2020], then the stacked neural network named DAE as Detector-AutoEncoder (https://gitlab.huma-num.fr/ANR-ROIi).
DAE was tested on a dataset of images synthetized from real vignettes randomly selected in two arbitrary classes, normals and abnormals, and placed in the image area without any typographical composition to properly challenge the model (https://documentation.huma-num.fr/nakala/). Normalized cross-correlation index between an original vignette and the reconstructed one shows a clear separation between normal and abnormal populations (Fig. 2). DAE has now to be compared with the state-of-the-art algorithms.
Figure 2. Histograms of the normalized cross-correlation index between an original vignette and the reconstructed one for abnormal and normal vignettes.
Rémi Emonet, Mohammad Sadil Khan, and Thierry Fournel
LHC-Laboratoire Hubert Curien-Saint-Etienne
Khan M. S., Emonet R. et Fournel T., Vignette detection and reconstruction of composed ornaments with a strengthened autoencoder, https://hal.archives-ouvertes.fr/hal-03409930, 2021
Bahier-Porte C. et Vial-Bonacci F, Le commerce de la librairie à la lumière de la correspondance – Marc Michel Rey, Pierre Rousseau, Charles Weissenbruch, Journal encyclopédique aux humanités numériques. Trois siècles d’histoire du livre et de la pensée à travers le Fonds Weissenbruch, Bruxelles, Archives générales du Royaume, p. 205-222, 2019
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., Berg, A. C., SSD: Single-Shot Multibox detector, in European conference on computer vision. Springer, Cham, p. 21-37, 2016
He, K., Gkioxari, G., Doll´ar, P., Girshick, R., Mask R-CNN, in Proceedings of the IEEE international conference on computer vision. p. 2961-2969, 2017
Zheng, Z., Wang, P., Liu, W., Li, J., Ye, R., Ren, D., Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression, in Proceedings of the AAAI Conference on Artificial Intelligence, p. 12993-13000, 2020