Integrating Multimodality and Context to Automatic Language Analysis
Create and release your Profile on Zintellect – Postdoctoral applicants must create an account and complete a profile in the on-line application system. Please note: your resume/CV may not exceed 3 pages.
Complete your application – Enter the rest of the information required for the IC Postdoc Program Research Opportunity. The application itself contains detailed instructions for each one of these components: availability, citizenship, transcripts, dissertation abstract, publication and presentation plan, and information about your Research Advisor co-applicant.
Additional information about the IC Postdoctoral Research Fellowship Program is available on the program website located at: https://orise.orau.gov/icpostdoc/index.html.
If you have questions, send an email to ICPostdoc@orau.org. Please include the reference code for this opportunity in your email.
Research Topic Description, including Problem Statement:
There are two main underlying problems with automatic approaches to language analysis: a lack of ability to account for context, and a lack of interpretability of language across different modalities (for example, audio, image, video, and text).
Human communication is exceedingly context dependent. As a simplified example, if I state that the table needs to be moved a listener will automatically use context clues to indicate whether I mean an item of furniture, or an excel style table. These might be physical context clues, or indicators from the co-text. When people are talking about sensitive, taboo, or illegal topics, this reliance on context increases even more. Automatic language tools are improving at utilizing co-text to help improve the accuracy of work, but they are still limited in the range of context that can be considered.
Online communications are exceedingly important to the intelligence community, and increasingly multimodal. This might be a soundtrack which changes the intended meaning of a picture (for example a classic circus soundtrack over a social media video of me parking my car, indicating that I am not showcasing my excellent parking skills but encouraging ridicule), or an emoji pasted over the top of an image (for example a picture of snow, with a nose emoji, indicating that the post is about nasally inhaled drugs rather than snow). Cross-modal communication like this is now the norm in many groups and societies, and that is particularly the case when discussing taboo (or illegal) topics.
Dover (2022) highlights how significant the internet and electronic communications are to intelligence communities. Automated approaches to language analysis can enable the quick triage and handling of significant amounts of data, however where they struggle significantly is with bringing together meaning from across different modes. This means that a significant amount of the communicative content risks being lost before it reaches an analyst.
These changes in meaning provided by either the context or the different modalities might be instantly understandable to us as humans, but an automated approach that struggles to consider such aspects, will provide a severely limited output. The topic here is designed to seek ways to combat these two problems – to integrate a holistic understanding of language with automatic language approaches. The desire is that the outputs will therefore be grounded in applied and sociolinguistics and able to address language and communication in a more accurate and reliable way, considering how language actually functions.
Example Approaches:
The exact approach will depend on the form of automatic language analyses that are being considered, though researchers will need to source their own data set(s) to show a proof of concept. An overarching example approach would be starting from a sociolinguistic or corpus linguistic perspective and seeking to ensure that the understanding of how language works remains in the automated approaches. This is supported by the literature, most notably Grieve et al. (2024), who in their recent paper on the Sociolinguistic Foundations of Language Modeling conclude that “incorporating insights from sociolinguistics is crucial to the future of language modeling” (p17).
However, the benefit of such integration has a much longer trail of evidence. For example, in a 2013 Native Language Identification challenge (sometimes called Other or Native Language Influence Detection), where participants seek to identify an author’s first language (when they are writing in English), Bykh et al. (2013) achieved a higher classification accuracy that other participants through using linguistically informed features in their classifier. This included features such as parts of speech, lemma realizations and use of derivational and inflectional suffixes. Further work on Other Language Influence Detection for forensic linguistic purposes by Kredens, Perkins, and Grant (2019) highlights how vital an explanatory rich approach (such as one grounded in sociolinguistic explanations and features) is to analysis of language in evidential and investigative situations.
Focusing on the concurrent analysis of both the verbal and visual aspects of Instagram posts at the same time, Caple (2018) shows that taking a corpus-assisted multimodal discourse analysis can reduce partiality and enable triangulation. Polli and Sindoni (2024) look at the multimodality in hateful memes, and that the interplay between non-hateful text and non-hateful images can be used to produce hateful messages. They note that multimodality is conceptualized differently across the domains of computer science and sociosemiotics, however they also show that AI driven models can benefit from sociosemiotic insights and incorporating a multimodal critical discourse analysis approach. More specific focused example approaches might include:
- Given a set of political speeches or news reports that happen over time in a changing context (e.g. during a conflict), how could an understanding of context improve topic modelling, document summarization, an understanding of the evolution of events, or other forms of automated analysis? This could include (for example) the change salience of different places or people during the conflict, or a need to show strength to an audience in reaction to provocation.
- Given a set of social media posts with associated images (e.g. memes), how could an understanding of meaning and mood be better extracted from the multi-media content? For example, memes such as Wojak and Pepe the frog are often adapted quickly to express emotion reactions and humor at a given situation – how could that data be analyzed alongside the text to give a more nuanced understanding of messages. Another example would be when images are used to convey instant emotional impact – for example in Daesh propaganda, CGI from video games was used to make it seem like the Eiffel tower had been attacked, or during the 2011 London riots images were shown of the London Eye on fire. Images like this may have more impact than just text messages.
Key Words: Linguistics, forensic linguistics, applied linguistics, computational linguistics,
Postdoc Eligibility
- U.S. citizens only
- Ph.D. in a relevant field must be completed before beginning the appointment and within five years of the appointment start date
- Proposal must be associated with an accredited U.S. university, college, or U.S. government laboratory
- Eligible candidates may only receive one award from the IC Postdoctoral Research Fellowship Program
Research Advisor Eligibility
- Must be an employee of an accredited U.S. university, college or U.S. government laboratory
- Are not required to be U.S. citizens
- Citizenship: U.S. Citizen Only
- Degree: Doctoral Degree.
-
Discipline(s):
- Chemistry and Materials Sciences (12 )
- Communications and Graphics Design (3 )
- Computer, Information, and Data Sciences (17 )
- Earth and Geosciences (21 )
- Engineering (27 )
- Environmental and Marine Sciences (14 )
- Life Health and Medical Sciences (45 )
- Mathematics and Statistics (11 )
- Other Non-Science & Engineering (2 )
- Physics (16 )
- Science & Engineering-related (1 )
- Social and Behavioral Sciences (30 )
![ORISE](/images/orise-logo-pdf.png)
![ORISE](/images/orise-go-logo.png)
![ORISE](/images/orise-go-phones.png)
The ORISE GO mobile app helps you stay engaged, connected and informed during your ORISE experience – from application, to offer, through your appointment and even as an ORISE alum!