You may have to increase the Java memory allocated to the gateway. A typical use case would be to upload a huge file using the Request: Configure Message Streaming assertion. 

This KB article outlines how to increase the Java memory allocated to the gateway.

All supported versions of the API Gateway

To check for how much memory is being allocated to Java for the gateway, use the ps -efa | grep Gateway.jar | grep --color Xmx command.

The following configuration file has the multiplication factor for the amount of memory. By default, it's 1/2.

/opt/SecureSpan/Gateway/runtime/etc/profile.d/appliancedefs.sh

Change the multiplication factor to a higher number than 1/2. For example, 2/3 or 4/5

REF: https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-gateway/10-1/reference/troubleshoot-the-gateway/optimize-gateway-for-large-file-uploads.html

Changing the Java memory may have a negative effect on the machine. Please try this on a test machine before production or contact Broadcom Support.