×

A new approach to image-based estimation of food volume. (English) Zbl 1461.62189

Summary: A balanced diet is the key to a healthy lifestyle and is crucial for preventing or dealing with many chronic diseases such as diabetes and obesity. Therefore, monitoring diet can be an effective way of improving people’s health. However, manual reporting of food intake has been shown to be inaccurate and often impractical. This paper presents a new approach to food intake quantity estimation using image-based modeling. The modeling method consists of three steps: firstly, a short video of the food is taken by the user’s smartphone. From such a video, six frames are selected based on the pictures’ viewpoints as determined by the smartphone’s orientation sensors. Secondly, the user marks one of the frames to seed an interactive segmentation algorithm. Segmentation is based on a Gaussian Mixture Model alongside the graph-cut algorithm. Finally, a customized image-based modeling algorithm generates a point-cloud to model the food. At the same time, a stochastic object-detection method locates a checkerboard used as size/ground reference. The modeling algorithm is optimized such that the use of six input images still results in an acceptable computation cost. In our evaluation procedure, we achieved an average accuracy of 92% on a test set that includes images of different kinds of pasta and bread, with an average processing time of about 23 s.

MSC:

62P10 Applications of statistics to biology and medical sciences; meta analysis
62H35 Image analysis in multivariate analysis

Software:

SURF; BRISK; PRMLT
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Mesas, A.; Muñoz-Pareja, M.; López-García, E.; Rodríguez-Artalejo, F.; Selected eating behaviours and excess body weight: A systematic review; Obes. Rev.: 2012; Volume 13 ,106-135.
[2] Livingstone, M.B.E.; Black, A.E.; Markers of the validity of reported energy intake; J. Nutr.: 2003; Volume 133 ,895S-920S.
[3] Schoeller, D.A.; Limitations in the assessment of dietary energy intake by self-report; Metabolism: 1995; Volume 44 ,18-22.
[4] Witschi, J.C.; Short-term dietary recall and recording methods; Nutr. Epidemiol.: 1990; Volume 4 ,52-68.
[5] Hassannejad, H.; Matrella, G.; Ciampolini, P.; De Munari, I.; Mordonini, M.; Cagnoni, S.; Automatic diet monitoring: A review of computer vision and wearable sensor-based methods; Int. J. Food Sci. Nutr.: 2017; . · Zbl 1461.62189
[6] Martin, C.K.; Kaya, S.; Gunturk, B.K.; Quantification of food intake using food image analysis; Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society: ; ,6869-6872.
[7] Xu, C.; He, Y.; Khannan, N.; Parra, A.; Boushey, C.; Delp, E.; Image-based food volume estimation; Proceedings of the 5th International Workshop on Multimedia for Cooking & Eating Activities: New York, NY, USA 2013; ,75-80.
[8] Almaghrabi, R.; Villalobos, G.; Pouladzadeh, P.; Shirmohammadi, S.; A novel method for measuring nutrition intake based on food image; Proceedings of the 2012 IEEE International Instrumentation and Measurement Technology Conference (I2MTC): ; ,366-370.
[9] Puri, M.; Zhu, Z.; Yu, Q.; Divakaran, A.; Sawhney, H.; Recognition and volume estimation of food intake using a mobile device; Proceedings of the 2009 IEEE Workshop on Applications of Computer Vision (WACV): ; ,1-8.
[10] Dehais, J.; Shevchik, S.; Diem, P.; Mougiakakou, S.G.; Food volume computation for self dietary assessment applications; Proceedings of the 2013 IEEE 13th International Conference on Bioinformatics and Bioengineering (BIBE): ; ,1-4.
[11] Hassannejad, H.; Matrella, G.; Mordonini, M.; Cagnoni, S.; Hassannejad, H.; A Mobile App for Food Detection: New approach to interactive segmentation; Proceedings of the FORITAAL Conference: ; ,306-313. · Zbl 1461.62189
[12] Hassannejad, H.; Matrella, G.; Mordonini, M.; Cagnoni, S.; Using Small Checkerboards as Size Reference: A Model-Based Approach; New Trends in Image Analysis and Processing-ICIAP 2015 Workshops: Cham, Switzerland 2015; ,393-400.
[13] Zhu, F.; Bosch, M.; Woo, I.; Kim, S.; Boushey, C.J.; Ebert, D.S.; Delp, E.J.; The use of mobile devices in aiding dietary assessment and evaluation; IEEE J. Sel. Top. Signal Process.: 2010; Volume 4 ,756-766.
[14] Dehais, J.; Anthimopoulos, M.; Mougiakakou, S.; Dish Detection and Segmentation for Dietary Assessment on Smartphones; New Trends in Image Analysis and Processing-ICIAP 2015 Workshops: Cham, Switzerland 2015; ,433-440.
[15] Eskin, Y.; Mihailidis, A.; An intelligent nutritional assessment system; In Proceedings of the 2012 AAAI Fall Symposium Series: Arlington, VA, USA ; .
[16] Zhu, F.; Bosch, M.; Khanna, N.; Boushey, C.J.; Delp, E.J.; Multiple hypotheses image segmentation and classification with application to dietary assessment; IEEE J. Biomed. Health Inf.: 2015; Volume 19 ,377-388.
[17] Kawano, Y.; Yanai, K.; Foodcam: A real-time mobile food recognition system employing fisher vector; MultiMedia Modeling: Cham, Switzerland 2014; ,369-373.
[18] Yanai, K.; Kawano, Y.; Food image recognition using deep convolutional network with pre-training and fine-tuning; Proceedings of the 2015 IEEE International Conference on Multimedia &: ; ,1-6.
[19] Oliveira, L.; Costa, V.; Neves, G.; Oliveira, T.; Jorge, E.; Lizarraga, M.; A mobile, lightweight, poll-based food identification system; Pattern Recognit.: 2014; Volume 47 ,1941-1952.
[20] Kohli, P.; Torr, P.H.; Dynamic graph cuts for efficient inference in markov random fields; IEEE Trans. Pattern Anal. Mach. Intell.: 2007; Volume 29 ,2079-2088.
[21] Bishop, C.; ; Pattern Recognition and Machine Learning: New York, NY, USA 2007; . · Zbl 1107.68072
[22] Rahman, M.H.; Li, Q.; Pickering, M.; Frater, M.; Kerr, D.; Bouchey, C.; Delp, E.; Food volume estimation in a mobile phone based dietary assessment system; Proceedings of the 2012 IEEE Eighth International Conference on Signal Image Technology and Internet Based Systems (SITIS): ; ,988-995.
[23] OpenCV; ; .
[24] Storn, R.; Price, K.; Differential evolution—A simple and efficient adaptive scheme for global optimization over continuous spaces; J. Glob. Optim.: 1995; Volume 3 ,341-359. · Zbl 0888.90135
[25] Ugolotti, R.; Nashed, Y.S.; Mesejo, P.; Ivekovič, Š.; Mussi, L.; Cagnoni, S.; Particle swarm optimization and differential evolution for model-based object detection; Appl. Soft Comput.: 2013; Volume 13 ,3092-3105.
[26] Lawson, C.L.; Hanson, R.J.; ; Solving Least Squares Problems: Philadelphia, PA, USA 1995; Volume Volume 15 . · Zbl 0860.65029
[27] Snavely, N.; Seitz, S.M.; Szeliski, R.; Modeling the world from internet photo collections; Int. J. Comput. Vis.: 2008; Volume 80 ,189-210.
[28] Baggio, D.L.; ; Mastering OpenCV with Practical Computer Vision Projects: Birmingham, UK 2012; .
[29] Bay, H.; Ess, A.; Tuytelaars, T.; Van Gool, L.; Speeded-up robust features (SURF); Comput. Vis. Image Underst.: 2008; Volume 110 ,346-359.
[30] Leutenegger, S.; Chli, M.; Siegwart, R.Y.; BRISK: Binary robust invariant scalable keypoints; Proceedings of the 2011 IEEE International Conference on Computer Vision: ; ,2548-2555.
[31] Hartley, R.; Zisserman, A.; ; Multiple View Geometry in Computer Vision: Cambridge, UK 2003; . · Zbl 0956.68149
[32] Raguram, R.; Frahm, J.M.; Pollefeys, M.; A comparative analysis of RANSAC techniques leading to adaptive real-time random sample consensus; Proceedings of the 10th European Conference on Computer Vision: Berlin, Germany 2008; ,500-513.
[33] Hartley, R.I.; Sturm, P.; Triangulation; Comput. Vis. Image Underst.: 1997; Volume 68 ,146-157.
[34] Triggs, B.; McLauchlan, P.F.; Hartley, R.I.; Fitzgibbon, A.W.; Bundle adjustment—A modern synthesis; Vision Algorithms: Theory and Practice: Berlin, Germany 1999; ,298-372.
[35] Qhull; ; .
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.