You cannot just download files.
If you prevent users from viewing files in the user rights settings, you will not be able to view or download files.
For details, refer to Setting Up Access Permissions for Folders.