Korean, Chinese, Japanese, and Arabic are rated as the hardest languages in the world for native speakers of English. How hard a language is depends on many factors including competency at learning foreign languages and especially the mother tongue of the person learning it. For example, English speakers can learn a language like Norwegian fairly easily because of how closely they are related, however a Chinese or Japanese person would struggle with it for much longer.
The US Government rates Japanese as harder than Chinese, and the hardest for native speakers of English, and being able to speak both I would agree. Japanese is subject-object-verb, has politeness levels which don't exist in English, and is an agglutinative language with many conjugations and inflections. Chinese on the other hand, is subject-verb-object like English, has no genders no conjugations no inflections no irregular verbs no politeness levels ,etc.
Chinese is only rated so difficult because of it's characters and tones, the latter is not as big of a deal as the former. Japanese has characters as well, and although there aren't as many as there in Chinese they are used in a more haphazard way. The readings for Chinese characters stay fairly constant, whereas in Japanese the reading for the character depends on how and where it's used in the sentence and can up to 6-8 different readings. Japanese also uses a combination of two scripts, hiragana and katakana, in addition to characters, whereas in Chinese it's just characters.
Korean is hard because it difficult grammar and things like politeness levels similar to Japanese, but with more complex pronunciation (but actually has an alphabet unlike Chinese and Japanese which have characters). I'm not so sure about what makes Arabic hard, I don't speak it.
I'm a linguist and also speak Chinese and Japanese fluently
Here is an assessment compiled by the Foreign Service Institute of the Department of State that lists the varying degrees of difficulty for various languages for native speakers of English