Speeding up data import to Neo4j v5 and CSV format data

[nickzren]

I encountered challenges while trying to load Hetionet data into my updated MacBook's Neo4j version 5.13. The existing Neo4j dumps were no longer compatible, and directly importing the data in JSON format was too time-consuming, taking an estimated 10+ hours.

To address this, I've written a script that efficiently converts JSON data to CSV format without any loss in node, edge, or property value information. The JSON-to-CSV conversion takes approximately 30 seconds, while uploading the CSV to Neo4j takes around 40 seconds.

I've organized each node and edge type into its own respective CSV file and accompanying Cypher script. I believe this will make it easier for people to understand and work with the data.

If this sounds useful, I'd be open to integrating these changes into the main branch. Let me know your thoughts.

You can find the revised code at:
https://github.com/nickzren/hetionet/tree/csv

[dhimmel]

Awesome work @nickzren. Nice job finding an efficient import method that works with the latest Neo4j stack.

I took a quick look at the changes and I'll need a little more time to think about where the code belongs... since it could possibly live in dhimmel/integrate or hetio/hetnetpy rather than in this repo whose focus is more the data and not the code to generate the data.

Taking a step back, there's a couple contributions that will be of major utility (in order of importance/interest):

1. a neo4j dump file that is compatible with neo4j 5 (and possibly future neo4j versions)
2. code to generate the neo4j database and dump file (i.e. your csv branch)
3. the csv files, but I'm a little cautious in that they have some similarities with the TSV files and we'd want to understand and document the differences.

Any thoughts?

[nickzren]

Thanks @dhimmel

I recognize the primary aim of this repo is data-centric rather than code-centric. However, I respectfully suggest that including data-specific scripts, such as Python and Cypher, could add value. No strong opinion here, and I'll defer to your ultimate decision on the matter.

The CSV data is derived from JSON files and serves as a comprehensive dataset, including all properties for both nodes and edges.

CSV files and Cypher scripts are generated simultaneously to ensure data type consistency, making the data import into Neo4j unaffected by version changes.

Having separate CSV files for nodes and edges not only enhances the framework's comprehensibility but also allows users to easily choose or modify or extend the data.

I encountered difficulties with restoring from Neo4j dump files and couldn't resolve the issues, so I began exploring alternative solutions.

I'm still learning about this and graph databases, so any corrections are welcome.

[sbonner0]

Hi @nickzren sorry for for messaging on this old issue - i've been trying to build a neo4j 5.x version of Hetionet for ages without luck.

I was happy to see you have done the hard work and have even pushed a new docker container. However, I've been trying to use it and I do not see any data available in the browser. Any idea why this might be? I can see there is data in the correct directory on the container itself. I used this command to start the docker:

docker run -d --name het5 \
  -p 7474:7474 -p 7687:7687 \
  nickzren/hetionet:latest

[nickzren]

@sbonner0 you can clone this repo https://github.com/nickzren/hetionet then run bash init.sh.

[sbonner0]

@nickzren Thanks for this - I can confirm that it worked now! Note that I had to clone with git lfs in order to get this to work properly.

I would be great if the pre-populated docker with all the data could be made available on dockerhub directly. I guess this would need to be discussed with Daniel.

[sbonner0]

Just out of interest I was also able to change your setup to build a version of the latest neo4j release: 2025.10

In the Dockerfile I changed the first line to:

FROM neo4j:2025.10

and I changed the compose file to build the container each time:

services:
  hetionet:
    build: .
    container_name: hetionet
    ports:
      - "7474:7474"
      - "7687:7687"

Thanks again for your work on this - really saved me a lot of time.

[nickzren]

@sbonner0 This is a good starting base, but it's better to figure out how to set up a data pipeline, make schema changes or adjustments based on your needs, and have the most up-to-date data loaded. Also check this repo: https://github.com/nickzren/PrimeKG