Comparison with Open Source MLIP

Accuracy of Calculations for Physical Properties

MLIPs are trained to predict atomic forces and system energies. However, from a practical standpoint in materials development, what truly matters is how accurately they can predict real-world physical properties. Therefore, when evaluating the performance of a universal MLIP, particularly one intended for application across a wide range of materials, it is essential to assess not only the accuracy of forces and energies, but also its ability to predict key physical properties. In this section, we compare PFP and OSS-MLIPs in terms of two properties critical for practical use: lattice thermal conductivity and surface energy.

Comparison of accuracy of lattice thermal conductivity

Lattice thermal conductivity is not only an important property in engineering, but also a stringent benchmark for evaluating MLIPs, as its calculation requires high-order derivatives of the potential energy surface. In this study, we predicted lattice thermal conductivity using PFP and several OSS-MLIPs, and compared the results with DFT data from the PhononDB-PBE dataset. Prediction errors were quantified using the mean Symmetric Relative Error (mSRE) for lattice thermal conductivity and the mean Symmetric Relative Mean Error (mSRME) for individual phonons—where lower values indicate higher accuracy. Table 1 summarizes the results. PFP demonstrated higher accuracy than other OSS-MLIPs in predicting lattice thermal conductivity.

Table 1. Comparison of lattice thermal conductivity prediction accuracy between PFP and OSS-MLIP

	mSRE (↓)	mSRME (↓)
PFP v6 (distance 0.03 Å)	0.530	0.656
PFP v6 (distance 0.1 Å)	0.245	0.374
MACE-L (Ref.)	0.710	0.935
MACE-L (distance 0.03 Å)	0.719	0.932
MACE-L (distance 0.1 Å)	0.694	0.915
MatterSim-v1 (Ref.)	0.413	0.575
MatterSim-v1 (distance 0.03 Å)	0.413	0.575
MatterSim-v1 (distance 0.1 Å)	0.366	0.541

For a detailed analysis, please see our Lattice Thermal Conductivity Calculation.

Surface energy accuracy comparison

In practical materials development, researchers work not only with ideal crystal structures of inorganic materials, but also with more complex configurations such as surfaces and interfaces. These structures often exhibit unique energetic characteristics, making surface energy a critical physical property. In this study, we predicted surface energies using PFP and OSS-MLIPs, and compared the results to DFT values from the CHIPS-FF Surface Energy dataset. The results, shown in Table 2, demonstrate that PFP achieves surface energy predictions with accuracy comparable to the best-performing methods.

Table 2. Accuracy of surface energy predictions by PFP and OSS-MLIP

Model name	MAE (J/m²)
PFP v7	0.19
eqV2_31M_omat_mp_salex	0.17
eqV2_31M_omat	0.18
eqV2_86M_omat_mp_salex	0.18
eqV2_153M_omat	0.19
eqV2_86M_omat	0.20
orb-v2	0.18
MatterSim-v1	0.36

Check out our Surface Energy Benchmarks for more information.

Calculation Number of Atoms and Speed Comparison

Modeling key phenomena in real-world materials development—such as reactions and diffusion at interfaces—requires more than just tracking the behavior of a small number of atoms. Therefore, when evaluating the performance of an MLIP, the number of atoms it can simulate and its computational speed are critical performance metrics.

Here, we present a comparison of PFP and five other MLIPs in terms of the maximum number of atoms that can be simulated and computational speed. PFP was executed on the Matlantis platform, while the other MLIPs were run using an NVIDIA A100 GPU (80 GB memory) with the ASE Calculator. For each method, calculation speed was measured as the average time required to compute atomic forces over 10 iterations.

As a result, PFP was able to handle 3 to 20 times more atoms than other OSS-MLIPs, and also achieved the fastest computation times when simulating the same number of atoms. These results suggest that PFP enables large-scale simulations that are impractical with OSS-MLIPs, as well as simulations of complex phenomena that were previously beyond the reach of conventional methods. This capability is expected to accelerate both the understanding of complex materials behavior and the accurate prediction of physical properties.