Can I Hide Graphs per User or per Department?

You cannot hide graphs added by administrators. However, you can set permissions to records/fields on a user or department basis. By doing so, the records and fields for which users do not have permissions will not be displayed on the summarized result.

Only the records/fields for which the user has view permission are summarized.
If users specify the field for which they do not have view permission in the sum settings, it will be excluded from the summarizing process and will not be displayed in the summarized result as well.