Is it possible to limit file access to a single user?

Yes, it requires setup. Access limitation can be setup using WordPress user roles (admin, author...)

• For each category of files you can setup witch user role will be able to display and download files
• For each file management you can setup action on user roles, for example authors can only manage their own files