I now spent some time in optimizing my mail server configuration, especially adjusting the rules, which clients and mails get accepted and which not. In its current setup, it's already configured to not act as an open relay. Mails sent via the server have to use SMTP-AUTH.
So these are the rules, that get applied now:
smtpd_delay_reject = yes smtpd_data_restrictions = reject_unauth_pipelining smtpd_client_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_invalid_hostname, reject_unknown_client_hostname, reject_non_fqdn_hostname, reject_rbl_client sbl-xbl.spamhaus.org smtpd_sender_restrictions = reject_unknown_address, reject_unknown_sender_domain, reject_non_fqdn_sender smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, reject_unknown_recipient_domain, reject_unverified_recipient, reject_non_fqdn_recipient, permit
At the moment it already reduced the amount of spam by around 90 %. I hve not yet seen any downside, but I have to check these settings a bit longer to be sure.