Is it possible to limit file access to a single user?
Yes, it requires setup. Access limitation can be setup using WordPress user roles (admin, author...)
- For each category of files you can setup witch user role will be able to display and download files
- For each file management you can setup action on user roles, for example authors can only manage their own files