Dalarna University's logo and link to the university's website

du.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • chicago-author-date
  • chicago-note-bibliography
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
A multimodal deep learning approach for gravel road condition evaluation through image and audio integration
Dalarna University, School of Information and Engineering, Microdata Analysis. (Microdata Analysis)ORCID iD: 0000-0003-2972-3635
Dalarna University, School of Information and Engineering, Statistics. (Microdata Analysis)ORCID iD: 0000-0002-3183-3756
Dalarna University, School of Information and Engineering, Informatics. (Microdata Analysis)ORCID iD: 0000-0003-4812-4988
2024 (English)In: Transportation Engineering, ISSN 2666-691X, Vol. 16, article id 100228Article in journal (Refereed) Published
Sustainable development
SDG 3: Good health and well-being, SDG 11: Sustainable cities and communities
Abstract [en]

This study investigates the combination of audio and image data to classify road conditions, particularly focusingon loose gravel scenarios. The dataset underwent binary categorisation, comprising audio segments capturinggravel sounds and corresponding images. Early feature fusion, utilising a pre-trained Very Deep ConvolutionalNetworks 19 (VGG19) and Principal component analysis (PCA), improved the accuracy of the Random Forestclassifier, surpassing other models in accuracy, precision, recall, and F1-score. Late fusion, involving decisionlevelprocessing with logical disjunction and conjunction gates (AND and OR) in combination with individualclassifiers for images and audio based on Densely Connected Convolutional Networks 121 (DenseNet121),demonstrated notable performance, especially with the OR gate, achieving 97 % accuracy. The late fusionmethod enhances adaptability by compensating for limitations in one modality with information from the other.Adapting maintenance based on identified road conditions minimises unnecessary environmental impact. Thismethod can help to identify loose gravel on gravel roads, substantially improving road safety and implementing aprecise maintenance strategy through a data-driven approach.

Place, publisher, year, edition, pages
Elsevier, 2024. Vol. 16, article id 100228
Keywords [en]
Gravel road maintenance, Data fusion, Sound analysis, Machine vision, Machine, Learning
National Category
Architectural Engineering
Identifiers
URN: urn:nbn:se:du-48032DOI: 10.1016/j.treng.2024.100228Scopus ID: 2-s2.0-85184492304OAI: oai:DiVA.org:du-48032DiVA, id: diva2:1837470
Available from: 2024-02-13 Created: 2024-02-13 Last updated: 2024-06-07
In thesis
1. Objective Assessment of Loose Gravel Condition using Machine Learning with Audio-visual Observation
Open this publication in new window or tab >>Objective Assessment of Loose Gravel Condition using Machine Learning with Audio-visual Observation
2024 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

A well-maintained road network is essential for sustainable economic development, providing vital transportation routes for goods and services while connecting communities. Sweden's public road network includes a significant portion of gravel roads, particularly cost-effective for less populated areas with lower traffic volumes. However, gravel roads deteriorate quickly, leading to accidents, environmental pollution, and vehicle tire wear when not adequately maintained. The Swedish Road Administration Authority (Trafikverket) assesses gravel road conditions using subjective methods, analysing images taken during snow-free periods. Due to cost constraints, this labour-intensive process is prone to errors and lacks advanced techniques like road profilometers.

This thesis explores the field of assessing gravel road conditions. It commences with a comprehensive review of manual gravel road assessment methods employed globally and existing data-driven smart methods. Subsequently, it harnesses machine hearing and machine vision techniques, primarily focusing on enhancing road condition classification by integrating sound and image data.

The research examines sound data collected from gravel roads, exploring machine learning algorithms for loose gravel conditions classification with potential road maintenance and monitoring implications. Another crucial aspect involves applying machine vision to categorise image data from gravel roads. The study introduces an innovative approach using publicly available resources like Google Street View for image data collection, demonstrating machine vision's adaptability in assessing road conditions.

The research also compares machine learning methods with manual human classification, specifically regarding sound data. Automated approaches consistently outperform manual methods, providing more reliable results. Furthermore, the thesis investigates combining audio and image data to classify road conditions, particularly loose gravel scenarios. Early feature fusion using pre-trained models significantly improves classifier accuracy.

The research proposes using cost-effective devices like mobile phones with AI applications attached to car windshields to collect audio and visual data on gravel road conditions. This approach can provide more accurate and efficient data collection, resulting in real-time mapping of road conditions over considerable distances. Such information can benefit drivers, travellers, and road maintenance agencies by identifying problematic areas with loose gravel, enabling targeted and efficient maintenance efforts, and minimising disruptions to traffic flow during maintenance operations.

Place, publisher, year, edition, pages
Borlänge: Dalarna University, 2024
Series
Dalarna Doctoral Dissertations ; 29
Keywords
Gravel road condition assessment, Loose gravel, Sound analysis, Machine learning, Image analysis, Audio analysis, Image and audio data fusion
National Category
Infrastructure Engineering Reliability and Maintenance
Identifiers
urn:nbn:se:du-47915 (URN)978-91-88679-59-8 (ISBN)
Public defence
2024-03-28, room Clas Ohlson, campus Borlänge, 13:00 (English)
Opponent
Supervisors
Available from: 2024-02-23 Created: 2024-01-25 Last updated: 2024-02-23Bibliographically approved

Open Access in DiVA

fulltext(3821 kB)75 downloads
File information
File name FULLTEXT01.pdfFile size 3821 kBChecksum SHA-512
b59fd4a567487b36536b97164e26966c1dea9b84bea9bfaef90492c08415af389acb2b71571f2d5ea268becbbdf441ebcb587dc73505cae5518d051ec1e3314d
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopushttps://www.sciencedirect.com/journal/transportation-engineering

Authority records

Saeed, NausheenAlam, MoududNyberg, Roger G.

Search in DiVA

By author/editor
Saeed, NausheenAlam, MoududNyberg, Roger G.
By organisation
Microdata AnalysisStatisticsInformatics
Architectural Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 76 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 126 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • chicago-author-date
  • chicago-note-bibliography
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf