Despite the impressive performance of Large Language Models (LLM) for various natural language processing tasks, little is known about their comprehension of geographic data and related ability to facilitate informed geospatial decision-making. This paper investigates the extent of geospatial knowledge, awareness, and reasoning abilities encoded within such pretrained LLMs. With a focus on autoregressive language models, we devise experimental approaches related to (i) probing LLMs for geo-coordinates to assess geospatial knowledge, (ii) using geospatial and non-geospatial prepositions to gauge their geospatial awareness, and (iii) utilizing a multidimensional scaling (MDS) experiment to assess the models’ geospatial reasoning capabilities and to determine locations of cities based on prompting. Our results confirm that it does not only take larger, but also more sophisticated LLMs to synthesize geospatial knowledge from textual information. As such, this research contributes to understanding the potential and limitations of LLMs in dealing with geospatial information.
@article{bhandari2023large,title={Are Large Language Models Geospatially Knowledgeable?},author={Bhandari, Prabin and Anastasopoulos, Antonios and Pfoser, Dieter},journal={arXiv preprint arXiv:2310.13002},year={2023},}
Trustworthiness of Children Stories Generated by Large Language Models
Prabin Bhandari, and Hannah Brennan
In Proceedings of the 16th International Natural Language Generation Conference, Sep 2023
Large Language Models (LLMs) have shown a tremendous capacity for generating literary text. However, their effectiveness in generating children’s stories has yet to be thoroughly examined. In this study, we evaluate the trustworthiness of children’s stories generated by LLMs using various measures, and we compare and contrast our results with both old and new children’s stories to better assess their significance. Our findings suggest that LLMs still struggle to generate children’s stories at the level of quality and nuance found in actual stories.
@inproceedings{bhandari-brennan-2023-trustworthiness,title={Trustworthiness of Children Stories Generated by Large Language Models},author={Bhandari, Prabin and Brennan, Hannah},booktitle={Proceedings of the 16th International Natural Language Generation Conference},month=sep,year={2023},address={Prague, Czechia},publisher={Association for Computational Linguistics},pages={352--361},}
