The Compiler Plugin accepts configurations for meminitial and maxmem. You can follow the example below to set the initial memory size to 128MB and the maximum memory usage to 512MB:
<project>
  [...]
  <build>
    [...]
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.0</version>
        <configuration>
          <fork>true</fork>
          <meminitial>128m</meminitial>
          <maxmem>512m</maxmem>
        </configuration>
      </plugin>
    </plugins>
    [...]
  </build>
  [...]
</project>