Is it possible to limit file access to a single user?

Yes, it requires setup. Access limitation can be setup using WordPress user roles (admin, author...)

  • For each category of files you can setup witch user role will be able to display and download files
  • For each file management you can setup action on user roles, for example authors can only manage their own files