Large language models (LLMs) are ubiquitous in modern day natural language processing. However, previous work has shown degraded LLM performance for under-represented English dialects. We analyze the effects of typifying “standard” American English language questions as non-”standard” dialectal variants on multiple choice question answering tasks and find up to a 20% reduction in accuracy. Additionally, we investigate the grammatical basis of under-performance in non-”standard” English questions. We find that individual grammatical rules have varied effects on performance, but some are more… Read More
Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks
Large language models (LLMs) are ubiquitous in modern day natural language processing. However, previous work has shown degraded LLM performance for under-represented English dialects. We analyze the effects of typifying “standard” American English language questions as non-”standard” dialectal variants on multiple choice question answering tasks and find up to a 20% reduction in accuracy. Additionally, we investigate the grammatical basis of under-performance in non-”standard” English questions. We find that individual grammatical rules have varied effects on performance, but some are more…
Related Posts
Inductive Domain Transfer In Misspecified Simulation-Based Inference
Simulation-based inference (SBI) is a statistical inference approach for estimating latent parameters of a physical…
4 tips for using Nano Banana to create amazing images
Get the scoop on how to use Nano Banana, the Gemini app's viral new image…
Leveling up your developer experience in Google AI Studio
AI Studio has released new updates that give developers even more control.