Some profiling has been done already which has identified that:
- A large number of strings/char are being held onto during indexing. I think it's safe to assume that's related to JS inferencing, but that will need to be confirmed. Once that is verified, we need to look into why that is happening.
- There are a huge number of disk writes occurring during JS inferencing. I confirmed this by re-indexing the referenced project with JS indexing on and off. We know the issue is coming from JS inferencing due to the fact that Index is not designed to support reads when doing lots of writes. We will need to determine how we can avoid or how we can minimize reading from the Index during inferencing.