GraphScope: A One-Stop Large-Scale Graph Computing System from Alibaba

Overview

graphscope-logo

A One-Stop Large-Scale Graph Computing System from Alibaba

GraphScope CI Coverage Playground Artifact HUB Docs-en FAQ-en Docs-zh FAQ-zh README-zh

GraphScope is a unified distributed graph computing platform that provides a one-stop environment for performing diverse graph operations on a cluster of computers through a user-friendly Python interface. GraphScope makes multi-staged processing of large-scale graph data on compute clusters simple by combining several important pieces of Alibaba technology: including GRAPE, MaxGraph, and Graph-Learn (GL) for analytics, interactive, and graph neural networks (GNN) computation, respectively, and the vineyard store that offers efficient in-memory data transfers.

Visit our website at graphscope.io to learn more.

Getting Started

We provide a Playground with a managed JupyterLab. Try GraphScope straight away in your browser!

GraphScope can run on clusters managed by Kubernetes within containers. For quickly getting started, we can set up a local Kubernetes cluster and take advantage of pre-built Docker images as follows.

Prerequisites

To run GraphScope on your local computer, the following dependencies or tools are required.

  • Docker
  • Python >= 3.6 (with pip)
  • Local Kubernetes cluster set-up tool (e.g. Kind)

On macOS, you can follow the official guides to install them and enable Kubernetes in Docker. For Ubuntu/CentOS Linux distributions, we provide a script to install the above dependencies and prepare the environment. On Windows, you may want to install Ubuntu on WSL2 to use the script.

# run the environment preparing script.
./scripts/prepare_env.sh

Installation

GraphScope client is distributed as a python package and can be easily installed with pip.

pip3 install graphscope

Note that graphscope requires pip>=19.0, if you meet error like "ERROR: Could not find a version that satisfies the requirement graphscope" please upgrade your pip with

pip3 install -U pip

Next, we will walk you through a concrete example to illustrate how GraphScope can be used by data scientists to effectively analyze large graphs.

Please note that we have not hardened this release for production use and it lacks important security features such as authentication and encryption, and therefore it is NOT recommended for production use (yet)!

Demo: Node Classification on Citation Network

ogbn-mag is a heterogeneous network composed of a subset of the Microsoft Academic Graph. It contains 4 types of entities(i.e., papers, authors, institutions, and fields of study), as well as four types of directed relations connecting two entities.

Given the heterogeneous ogbn-mag data, the task is to predict the class of each paper. Node classification can identify papers in multiple venues, which represent different groups of scientific work on different topics. We apply both the attribute and structural information to classify papers. In the graph, each paper node contains a 128-dimensional word2vec vector representing its content, which is obtained by averaging the embeddings of words in its title and abstract. The embeddings of individual words are pre-trained. The structural information is computed on-the-fly.

how-it-works

The figure shows the flow of execution when a client Python program is executed.

  • Step 1. Create a session or workspace in GraphScope.
  • Step 2. Define schema and load the graph.
  • Step 3. Query graph data.
  • Step 4. Run graph algorithms.
  • Step 5. Run graph-based machine learning tasks.
  • Step 6. Close the session.

Creating a session

To use GraphScope, we need to establish a session in a python interpreter.

import os
import graphscope

# assume we mount `~/test_data` to `/testingdata` in pods.
k8s_volumes = {
    "data": {
        "type": "hostPath",
        "field": {
            "path": os.path.expanduser("~/test_data/"),
            "type": "Directory"
        },
        "mounts": {
            "mountPath": "/testingdata"
        }
    }
}

sess = graphscope.session(k8s_volumes=k8s_volumes)

For macOS, the session needs to establish with the LoadBalancer service type (which is NodePort by default).

sess = graphscope.session(k8s_volumes=k8s_volumes, k8s_service_type="LoadBalancer")

A session tries to launch a coordinator, which is the entry for the back-end engines. The coordinator manages a cluster of resources (k8s pods), and the interactive/analytical/learning engines ran on them. For each pod in the cluster, there is a vineyard instance at service for distributed data in memory.

Loading a graph

GraphScope models graph data as property graph, in which the edges/vertices are labeled and have many properties. Taking ogbn-mag as example, the figure below shows the model of the property graph.

sample-of-property-graph

This graph has four kinds of vertices, labeled as paper, author, institution and field_of_study. There are four kinds of edges connecting them, each kind of edges has a label and specifies the vertex labels for its two ends. For example, cites edges connect two vertices labeled paper. Another example is writes, it requires the source vertex is labeled author and the destination is a paper vertex. All the vertices and edges may have properties. e.g., paper vertices have properties like features, publish year, subject label, etc.

To load this graph to GraphScope, one may use the code below with the data files. Please download and extract it to the mounted dir on local(in this case, ~/test_data).

g = sess.g()
g = g.add_vertices("/testingdata/ogbn_mag_small/paper.csv", label="paper")
g = g.add_vertices("/testingdata/ogbn_mag_small/author.csv", label="author")
g = g.add_vertices("/testingdata/ogbn_mag_small/institution.csv", label="institution")
g = g.add_vertices("/testingdata/ogbn_mag_small/field_of_study.csv", label="field_of_study")
g = g.add_edges(
    "/testingdata/ogbn_mag_small/author_affiliated_with_institution.csv",
    label="affiliated", src_label="author", dst_label="institution",
)
g = g.add_edges(
    "/testingdata/ogbn_mag_small/paper_has_topic_field_of_study.csv",
    label="hasTopic", src_label="paper", dst_label="field_of_study",
)
g = g.add_edges(
    "/testingdata/ogbn_mag_small/paper_cites_paper.csv",
    label="cites", src_label="paper", dst_label="paper",
)
g = g.add_edges(
    "/testingdata/ogbn_mag_small/author_writes_paper.csv",
    label="writes", src_label="author", dst_label="paper",
)

Alternatively, we provide a function to load this graph for convenience.

from graphscope.dataset.ogbn_mag import load_ogbn_mag

g = load_ogbn_mag(sess, "/testingdata/ogbn_mag_small/")

Here, the g is loaded in parallel via vineyard and stored in vineyard instances in the cluster managed by the session.

Interactive query

Interactive queries allow users to directly explore, examine, and present graph data in an exploratory manner in order to locate specific or in-depth information in time. GraphScope adopts a high-level language called Gremlin for graph traversal, and provides efficient execution at scale.

In this example, we use graph traversal to count the number of papers two given authors have co-authored. To simplify the query, we assume the authors can be uniquely identified by ID 2 and 4307, respectively.

# get the endpoint for submitting Gremlin queries on graph g.
interactive = sess.gremlin(g)

# count the number of papers two authors (with id 2 and 4307) have co-authored
papers = interactive.execute("g.V().has('author', 'id', 2).out('writes').where(__.in('writes').has('id', 4307)).count()").one()

Graph analytics

Graph analytics is widely used in real world. Many algorithms, like community detection, paths and connectivity, centrality are proven to be very useful in various businesses. GraphScope ships with a set of built-in algorithms, enables users easily analysis their graph data.

Continuing our example, below we first derive a subgraph by extracting publications in specific time out of the entire graph (using Gremlin!), and then run k-core decomposition and triangle counting to generate the structural features of each paper node.

Please note that many algorithms may only work on homogeneous graphs, and therefore, to evaluate these algorithms over a property graph, we need to project it into a simple graph at first.

# extract a subgraph of publication within a time range
sub_graph = interactive.subgraph("g.V().has('year', inside(2014, 2020)).outE('cites')")

# project the projected graph to simple graph.
simple_g = sub_graph.project(vertices={"paper": []}, edges={"cites": []})

ret1 = graphscope.k_core(simple_g, k=5)
ret2 = graphscope.triangles(simple_g)

# add the results as new columns to the citation graph
sub_graph = sub_graph.add_column(ret1, {"kcore": "r"})
sub_graph = sub_graph.add_column(ret2, {"tc": "r"})

In addition, users can write their own algorithms in GraphScope. Currently, GraphScope support users to write their own algorithms in Pregel model and PIE model.

Graph neural networks (GNNs)

Graph neural networks (GNNs) combines superiority of both graph analytics and machine learning. GNN algorithms can compress both structural and attribute information in a graph into low-dimensional embedding vectors on each node. These embeddings can be further fed into downstream machine learning tasks.

In our example, we train a GCN model to classify the nodes (papers) into 349 categories, each of which represents a venue (e.g. pre-print and conference). To achieve this, first we launch a learning engine and build a graph with features following the last step.

# define the features for learning
paper_features = []
for i in range(128):
    paper_features.append("feat_" + str(i))

paper_features.append("kcore")
paper_features.append("tc")

# launch a learning engine.
lg = sess.learning(sub_graph, nodes=[("paper", paper_features)],
                  edges=[("paper", "cites", "paper")],
                  gen_labels=[
                      ("train", "paper", 100, (0, 75)),
                      ("val", "paper", 100, (75, 85)),
                      ("test", "paper", 100, (85, 100))
                  ])

Then we define the training process, and run it.

# Note: Here we use tensorflow as NN backend to train GNN model. so please
# install tensorflow.
from graphscope.learning.examples import GCN
from graphscope.learning.graphlearn.python.model.tf.trainer import LocalTFTrainer
from graphscope.learning.graphlearn.python.model.tf.optimizer import get_tf_optimizer

# supervised GCN.

def train(config, graph):
    def model_fn():
        return GCN(
            graph,
            config["class_num"],
            config["features_num"],
            config["batch_size"],
            val_batch_size=config["val_batch_size"],
            test_batch_size=config["test_batch_size"],
            categorical_attrs_desc=config["categorical_attrs_desc"],
            hidden_dim=config["hidden_dim"],
            in_drop_rate=config["in_drop_rate"],
            neighs_num=config["neighs_num"],
            hops_num=config["hops_num"],
            node_type=config["node_type"],
            edge_type=config["edge_type"],
            full_graph_mode=config["full_graph_mode"],
        )
    trainer = LocalTFTrainer(
        model_fn,
        epoch=config["epoch"],
        optimizer=get_tf_optimizer(
            config["learning_algo"], config["learning_rate"], config["weight_decay"]
        ),
    )
    trainer.train_and_evaluate()


config = {
    "class_num": 349,  # output dimension
    "features_num": 130,  # 128 dimension + kcore + triangle count
    "batch_size": 500,
    "val_batch_size": 100,
    "test_batch_size": 100,
    "categorical_attrs_desc": "",
    "hidden_dim": 256,
    "in_drop_rate": 0.5,
    "hops_num": 2,
    "neighs_num": [5, 10],
    "full_graph_mode": False,
    "agg_type": "gcn",  # mean, sum
    "learning_algo": "adam",
    "learning_rate": 0.0005,
    "weight_decay": 0.000005,
    "epoch": 20,
    "node_type": "paper",
    "edge_type": "cites",
}

train(config, lg)

See more details in node_classification_on_citation.ipynb, with the running results. Please note that learning feature of GraphScope is not support on macOS yet.

Closing the session

At last, we close the session after processing all graph tasks.

sess.close()

This operation will notify the backend engines and vineyard to safely unload graphs and their applications, Then, the coordinator will dealloc all the applied resources in the k8s cluster.

Development

Building Docker images

GraphScope ships with a Dockerfile that can build docker images for releasing. The images are built on a builder image with all dependencies installed and copied to a runtime-base image. To build images with latest version of GraphScope, go to the root directory and run this command.

# for the first time, run this script to install make, doxygen for docs and java env for testing.
# ./scripts/prepare_dev.sh

make graphscope
make interactive_manager
# by default, the built image is tagged as graphscope/graphscope:SHORTSHA and graphscope/maxgraph_standalone_manager:SHORTSHA

Building client library

GraphScope python interface is separate with the engines image. If you are developing python client and not modifying the protobuf files, the engines image doesn't require to be rebuilt.

You may want to re-install the python client on local.

cd python
python3 setup.py install

Note that the learning engine client has C/C++ extensions modules and setting up the build environment is a bit tedious. By default the locally-built client library doesn't include the support for learning engine. If you want to build client library with learning engine enabled, please refer Build Python Wheels.

Testing

To verify the correctness of your developed features, your code changes should pass our tests.

You may run the whole test suite with commands:

# for the first time, run this script to install make, doxygen for docs and java env for testing.
# ./scripts/prepare_dev.sh

# run all test cases
./scripts/test.sh --all

# run all test cases on your built image
./scripts/test.sh --all --gs_image graphscope/graphscope:SHORTSHA --gie_manager_image graphscope/maxgraph_standalone_manager:SHORTSHA

# or run the selected cases on a certain module. e.g.,
./scripts/test.sh --python
./scripts/test.sh --gie

Documentation

Documentation can be generated using Sphinx. Users can build the documentation using:

# build the docs
make graphscope-docs

# to open preview on local
open docs/_build/html/index.html

The latest version of online documentation can be found at https://graphscope.io/docs

License

GraphScope is released under Apache License 2.0. Please note that third-party libraries may not have the same license as GraphScope.

Contributing

Any contributions you make are greatly appreciated!

  • Join in the Slack channel for discussion.
  • Please report bugs by submitting a GitHub issue.
  • Please submit contributions using pull requests.
Comments
  • [BUG] Create GIE failed using graphscope deployed by helm

    [BUG] Create GIE failed using graphscope deployed by helm

    Describe the bug deploy with helm on k8s

    To Reproduce

    1. loaded dataset success
    2. create GIE failed
    # Import the graphscope module
    import os
    import graphscope
    graphscope.set_option(show_log=False)  # enable logging
    graphscope.set_option(log_level='DEBUG') # log level
    sess = graphscope.session(addr='ip:port',cluster_type='k8s', mount_dataset='/home/data/testingdata')
    
    # Load the obgn_mag dataset as a graph
    from graphscope.dataset import load_ogbn_mag
    graph = load_ogbn_mag(sess,'/tmp/testingdata/')
    
    # get the entrypoint for submitting Gremlin queries on graph g.
    interactive = sess.gremlin(graph)
    print(interactive.graph_url)
    

    Expected behavior coodinator log Create GIE instance with command: /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/bin/giectl create_gremlin_instance_on_k8s /tmp/gs/graphscope-1666685425/session_pkzhchoo 105809399820113850 /tmp/graph_7LaSXd5t.json gs-engine-graphscope-1666685425-7mxj6,gs-engine-graphscope-1666685425-lqspj engine 8239 8240 8241 graphscope-1666685425-coordinator

    juputer log ``

    Screenshots If applicable, add screenshots to help explain your problem.

    Environment (please complete the following information):

    • GraphScope version: 0.17.0
    • OS: Linux
    • Version [e.g. 10.15]
    • Kubernetes Version 1.19.5

    Additional context Add any other context about the problem here.

    opened by JackyYangPassion 21
  • [BUG]

    [BUG] "python3 setup.py install" failed on mac os

    Describe the bug guilded by the instructions on https://graphscope.io/docs/zh/developer_guide.html#id2

    i had a successful "make graphscope"

    when i ran "python3 setup.py install"

    it failed as below

    ...... ......

    some warning message

    ...... ......

    third_party/protobuf/src/google/protobuf/compiler/parser.cc:99:21: warning: comparison of integers of different signs: 'int' and 'std::__1::basic_string::size_type' (aka 'unsigned long') [-Wsign-compare] for (int i = 0; i < field_name.size(); ++i) { ~ ^ ~~~~~~~~~~~~~~~~~ third_party/protobuf/src/google/protobuf/compiler/parser.cc:133:21: warning: comparison of integers of different signs: 'int' and 'std::__1::basic_string::size_type' (aka 'unsigned long') [-Wsign-compare] for (int i = 1; i < name.length(); i++) { ~ ^ ~~~~~~~~~~~~~ third_party/protobuf/src/google/protobuf/compiler/parser.cc:142:21: warning: comparison of integers of different signs: 'int' and 'std::__1::basic_string::size_type' (aka 'unsigned long') [-Wsign-compare] for (int i = 0; i < name.length(); i++) { ~ ^ ~~~~~~~~~~~~~ third_party/protobuf/src/google/protobuf/compiler/parser.cc:152:21: warning: comparison of integers of different signs: 'int' and 'std::__1::basic_string::size_type' (aka 'unsigned long') [-Wsign-compare] for (int i = 0; i < name.length(); i++) { ~ ^ ~~~~~~~~~~~~~ third_party/protobuf/src/google/protobuf/compiler/parser.cc:162:21: warning: comparison of integers of different signs: 'int' and 'std::__1::basic_string::size_type' (aka 'unsigned long') [-Wsign-compare] for (int i = 1; i < name.length(); i++) { ~ ^ ~~~~~~~~~~~~~ third_party/protobuf/src/google/protobuf/compiler/parser.cc:505:21: warning: comparison of integers of different signs: 'int' and 'std::__1::vector<std::__1::basic_string>::size_type' (aka 'unsigned long') [-Wsign-compare] for (int i = 0; i < detached_comments->size(); ++i) { ~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~ 6 warnings generated. Traceback (most recent call last): File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 148, in setup dist.run_commands() File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 172, in run cmd = self.call_command('install_lib', warn_dir=0) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 158, in call_command self.run_command(cmdname) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 15, in run self.byte_compile(outfiles) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/command/install_lib.py", line 132, in byte_compile byte_compile(files, optimize=0, File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/util.py", line 466, in byte_compile compile(file, cfile, dfile) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/py_compile.py", line 157, in compile os.makedirs(dirname) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/os.py", line 213, in makedirs makedirs(head, exist_ok=exist_ok) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/os.py", line 213, in makedirs makedirs(head, exist_ok=exist_ok) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/os.py", line 213, in makedirs makedirs(head, exist_ok=exist_ok) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/os.py", line 223, in makedirs mkdir(name, mode) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 310, in wrap path = self._remap_input(name, path, *args, **kw) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 452, in _remap_input self._violation(operation, os.path.realpath(path), *args, **kw) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 407, in _violation raise SandboxViolation(operation, args, kw) setuptools.sandbox.SandboxViolation: SandboxViolation: mkdir('/Users/alwaysdaylast/Library/Caches/com.apple.python/private/var/folders/0p/7fr9ymt173q7y98bk5c9tnt02rxv17/T/easy_install-prf1um98/grpcio-tools-1.39.0rc1/build', 511) {}

    The package setup script has attempted to modify files on your system that are not within the EasyInstall build area, and has been aborted.

    This package cannot be safely installed by EasyInstall, and may not support alternate installation locations even if you run its setup script by hand. Please inform the package's author and the EasyInstall maintainers to find out if a fix or workaround is available.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 154, in save_modules yield saved File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 195, in setup_context yield File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 250, in run_setup _execfile(setup_script, ns) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 45, in _execfile exec(code, globals, locals) File "/var/folders/0p/7fr9ymt173q7y98bk5c9tnt02rxv17/T/easy_install-prf1um98/grpcio-tools-1.39.0rc1/setup.py", line 285, in """ File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/init.py", line 145, in setup return distutils.core.setup(**attrs) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 163, in setup raise SystemExit("error: " + str(msg)) SystemExit: error: SandboxViolation: mkdir('/Users/alwaysdaylast/Library/Caches/com.apple.python/private/var/folders/0p/7fr9ymt173q7y98bk5c9tnt02rxv17/T/easy_install-prf1um98/grpcio-tools-1.39.0rc1/build', 511) {}

    The package setup script has attempted to modify files on your system that are not within the EasyInstall build area, and has been aborted.

    This package cannot be safely installed by EasyInstall, and may not support alternate installation locations even if you run its setup script by hand. Please inform the package's author and the EasyInstall maintainers to find out if a fix or workaround is available.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 1144, in run_setup run_setup(setup_script, args) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 253, in run_setup raise File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in exit self.gen.throw(type, value, traceback) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 195, in setup_context yield File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in exit self.gen.throw(type, value, traceback) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 166, in save_modules saved_exc.resume() File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 141, in resume six.reraise(type, exc, self._tb) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/_vendor/six.py", line 685, in reraise raise value.with_traceback(tb) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 154, in save_modules yield saved File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 195, in setup_context yield File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 250, in run_setup _execfile(setup_script, ns) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 45, in _execfile exec(code, globals, locals) File "/var/folders/0p/7fr9ymt173q7y98bk5c9tnt02rxv17/T/easy_install-prf1um98/grpcio-tools-1.39.0rc1/setup.py", line 285, in """ File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/init.py", line 145, in setup return distutils.core.setup(**attrs) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 163, in setup raise SystemExit("error: " + str(msg)) SystemExit: error: SandboxViolation: mkdir('/Users/alwaysdaylast/Library/Caches/com.apple.python/private/var/folders/0p/7fr9ymt173q7y98bk5c9tnt02rxv17/T/easy_install-prf1um98/grpcio-tools-1.39.0rc1/build', 511) {}

    The package setup script has attempted to modify files on your system that are not within the EasyInstall build area, and has been aborted.

    This package cannot be safely installed by EasyInstall, and may not support alternate installation locations even if you run its setup script by hand. Please inform the package's author and the EasyInstall maintainers to find out if a fix or workaround is available.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "setup.py", line 288, in setup( File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/init.py", line 144, in setup _install_setup_requires(attrs) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/init.py", line 139, in _install_setup_requires dist.fetch_build_eggs(dist.setup_requires) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/dist.py", line 716, in fetch_build_eggs resolved_dists = pkg_resources.working_set.resolve( File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pkg_resources/init.py", line 780, in resolve dist = best[req.key] = env.best_match( File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pkg_resources/init.py", line 1065, in best_match return self.obtain(req, installer) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pkg_resources/init.py", line 1077, in obtain return installer(requirement) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/dist.py", line 786, in fetch_build_egg return cmd.easy_install(req) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 679, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 705, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 890, in install_eggs return self.build_and_install(setup_script, setup_base) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 1158, in build_and_install self.run_setup(setup_script, setup_base, args) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 1146, in run_setup raise DistutilsError("Setup script exited with %s" % (v.args[0],)) distutils.errors.DistutilsError: Setup script exited with error: SandboxViolation: mkdir('/Users/alwaysdaylast/Library/Caches/com.apple.python/private/var/folders/0p/7fr9ymt173q7y98bk5c9tnt02rxv17/T/easy_install-prf1um98/grpcio-tools-1.39.0rc1/build', 511) {}

    The package setup script has attempted to modify files on your system that are not within the EasyInstall build area, and has been aborted.

    This package cannot be safely installed by EasyInstall, and may not support alternate installation locations even if you run its setup script by hand. Please inform the package's author and the EasyInstall maintainers to find out if a fix or workaround is available.

    Environment (please complete the following information):

    GraphScope version: [ master] OS: [ MacOS] Version [11.4] any advice will be appreciated, thanks in advance.

    bug component:python component:dev-infra 
    opened by chen-ziang 19
  • [BUG] graphscope.framework.errors.GRPCError: 'RPC failed, the engine might have crashed: rpc create_interactive_engine: failed with error code StatusCode.UNKNOWN, details: Exception calling application: <urlopen error [Errno 111] Connection refused>'

    [BUG] graphscope.framework.errors.GRPCError: 'RPC failed, the engine might have crashed: rpc create_interactive_engine: failed with error code StatusCode.UNKNOWN, details: Exception calling application: '

    Describe the bug

    graphscope.framework.errors.GRPCError: 'RPC failed, the engine might have crashed: rpc create_interactive_engine: failed with error code StatusCode.UNKNOWN, details: Exception calling application: <urlopen error [Errno 111] Connection refused>' To Reproduce

    Steps to reproduce the behavior: import os import graphscope from graphscope.framework.graph import Graph from graphscope.framework.loader import Loader import vineyard graphscope.set_option(show_log=True) # enable logging graphscope.set_option(initializing_interactive_engine=False) sess = graphscope.session() # create a session print(sess) graph = sess.g() print(graph) interactive = sess.gremlin(graph) print(interactive)

    Expected behavior graphscope session should be able to interact with GIE server .

    Screenshots If applicable, add screenshots to help explain your problem. I0729 14:14:31.375777 84 grape_instance.cc:110] Loading graph, graph name: graph_a37JncCH, graph type: ArrowFragment, type sig: e33529e80839a2064a804ce453c761a9483aa7ab775bcfddc1a1f9da63dcb521 I0729 14:14:31.372226 89 grape_instance.cc:110] Loading graph, graph name: graph_a37JncCH, graph type: ArrowFragment, type sig: e33529e80839a2064a804ce453c761a9483aa7ab775bcfddc1a1f9da63dcb521 I0729 14:14:32.171826 84 property_graph_frame.cc:107] [worker-1] loaded graph to vineyard ... I0729 14:14:32.180032 89 property_graph_frame.cc:107] [worker-0] loaded graph to vineyard ... graphscope.Graph ARROW_PROPERTY

    Traceback (most recent call last): File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/graphscope/client/rpc.py", line 47, in with_grpc_catch return fn(*args, **kwargs) File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/graphscope/client/rpc.py", line 169, in create_interactive_engine response = self._stub.CreateInteractiveInstance(request) File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/grpc/_channel.py", line 946, in call return _end_unary_response_blocking(state, call, False, None) File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNKNOWN details = "Exception calling application: <urlopen error [Errno 111] Connection refused>" debug_error_string = "{"created":"@1627539272.263514146","description":"Error received from peer ipv4:10.15.18.10:31281","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"Exception calling application: <urlopen error [Errno 111] Connection refused>","grpc_status":2}"

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last): File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/graphscope/client/session.py", line 1135, in gremlin engine_params=engine_params, File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/graphscope/client/rpc.py", line 54, in with_grpc_catch ) from exc graphscope.framework.errors.GRPCError: 'RPC failed, the engine might have crashed: rpc create_interactive_engine: failed with error code StatusCode.UNKNOWN, details: Exception calling application: <urlopen error [Errno 111] Connection refused>'

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last): File "test.py", line 12, in interactive = sess.gremlin(graph) File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/graphscope/client/utils.py", line 156, in wrapper return_value = func(*args, **kwargs) File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/graphscope/client/session.py", line 1140, in gremlin raise InteractiveEngineInternalError(str(e)) from e graphscope.framework.errors.InteractiveEngineInternalError: "'RPC failed, the engine might have crashed: rpc create_interactive_engine: failed with error code StatusCode.UNKNOWN, details: Exception calling application: <urlopen error [Errno 111] Connection refused>'" 2021-07-29 14:14:33,525 [INFO][coordinator:635]: Coordinator close interactive instance with url[http://xx.xx.xx.xx:8080/instance/close?graphName=11355625130038992&podNameList=gs-engine-juyknj-284dr,gs-engine-juyknj-bz2gf&containerName=engine&waitingForDelete=False] 2021-07-29 14:14:33,528 [ERROR][coordinator:639]: Failed to close interactive instance: <urlopen error [Errno 111] Connection refused> 2021-07-29 14:14:33,528 [ERROR][coordinator:639]: Failed to close interactive instance: <urlopen error [Errno 111] Connection refused>

    Environment (please complete the following information): GraphScope version: 0.5.0 OS: Ubuntu 20.04.2 Kubernetes Version v1.20.5 python3: version3.6.8

    Additional context Add any other context about the problem here.

    opened by 346057177 15
  • GraphScope Gremlin 引擎问题反馈

    GraphScope Gremlin 引擎问题反馈

    Describe the bug 使用 Gremlin 查询过程中遇到的问题,都反馈到这个 issues 下面, 麻烦 @yecol 统一看一下,感谢!

    • [x] nodejs id 不一致
    • [x] valueMap 支持
    • [x] gremlin 3.5.1 patch 包发布
    • [x] bothE failed
    • [x] both().bothE() failed
    • [x] valueMap 为 None 的点也返回
    • [ ] 边属性 (#1723 )
    • [x] 模式匹配:JSON 转 Gremlin 语句,查询结果
    • [ ] ~~g.V().has('id', 'xxx'),报错 key id must be positive number~~
    • [ ] repeat operator support (#1722 )
    priority:high component:gaia 
    opened by baizn 13
  • [BUG] Error running example code

    [BUG] Error running example code

    Describe the bug

    [error] Check failed: IOError: Receive message failed: Connection reset by peer in "client->Connect(vineyard_socket)", in function void gs::EnsureClient(std::shared_ptr<vineyard::Client>&, const string&), file /work/analytical_engine/core/launcher.cc, line 123
    terminate called after throwing an instance of 'std::runtime_error'
      what():  Check failed: IOError: Receive message failed: Connection reset by peer in "client->Connect(vineyard_socket)", in function void gs::EnsureClient(std::shared_ptr<vineyard::Client>&, const string&), file /work/analytical_engine/core/launcher.cc, line 123
    *** Aborted at 1659058982 (unix time) try "date -d @1659058982" if you are using GNU date ***
    PC: @                0x0 (unknown)
    *** SIGABRT (@0x3f00001ede5) received by PID 126437 (TID 0x7f813dbc1040) from PID 126437; stack trace: ***
        @     0x7f813fd223c0 (unknown)
        @     0x7f813ece818b gsignal
        @     0x7f813ecc7859 abort
        @     0x7f813f09c911 (unknown)
        @     0x7f813f0a838c (unknown)
        @     0x7f813f0a83f7 std::terminate()
        @     0x7f813f0a86a9 __cxa_throw
        @     0x7f8149263000 (unknown)
        @           0x48ebc0 gs::GrapeInstance::Init()
        @           0x46f353 gs::GrapeEngine::Start()
        @           0x45da42 main
        @     0x7f813ecc90b3 __libc_start_main
        @           0x45e345 (unknown)
        @                0x0 (unknown)
    Traceback (most recent call last):
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/runpy.py", line 194, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/gscoordinator/__main__.py", line 3, in <module>
        launch_graphscope()
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/gscoordinator/coordinator.py", line 1779, in launch_graphscope
        coordinator_service_servicer = CoordinatorServiceServicer(
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/gscoordinator/coordinator.py", line 175, in __init__
        if not self._launcher.start():
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/gscoordinator/launcher.py", line 174, in start
        self._create_services()
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/gscoordinator/launcher.py", line 610, in _create_services
        self._start_analytical_engine()
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/gscoordinator/launcher.py", line 592, in _start_analytical_engine
        time.sleep(1)
    KeyboardInterrupt
    Traceback (most recent call last):
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/graphscope/client/rpc.py", line 72, in waiting_service_ready
        self._stub.HeartBeat(request)
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
        return _end_unary_response_blocking(state, call, False, None)
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
        raise _InactiveRpcError(state)
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
            status = StatusCode.UNAVAILABLE
            details = "failed to connect to all addresses"
            debug_error_string = "{"created":"@1659059576.256340172","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3260,"referenced_errors":[{"created":"@1659059576.256338821","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}"
    >
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "gcn.py", line 11, in <module>
        graph = load_cora()
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/graphscope/dataset/cora.py", line 80, in load_cora
        sess = get_default_session()
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/graphscope/client/session.py", line 1477, in get_default_session
        return _default_session_stack.get_default()
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/graphscope/client/session.py", line 1498, in get_default
        sess = session(cluster_type="hosts", num_workers=1)
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/graphscope/client/utils.py", line 357, in wrapper
        return_value = func(*args, **kwargs)
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/graphscope/client/session.py", line 724, in __init__
        self._connect()
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/graphscope/client/session.py", line 1065, in _connect
        self._grpc_client.waiting_service_ready(
      File "/home/sanzo/software/miniconda/4.12/envs/sanzo/lib/python3.8/site-packages/graphscope/client/rpc.py", line 82, in waiting_service_ready
        raise ConnectionError(f"Connect coordinator timeout, {msg}")
    ConnectionError: Connect coordinator timeout, code: UNAVAILABLE, details: failed to connect to all addresses
    

    To Reproduce

    import graphscope
    from graphscope.dataset import load_ogbn_mag
    
    g = load_ogbn_mag()
    

    Environment (please complete the following information):

    • GraphScope version: 0.15.0
    • OS: Linux
    • Version: 20.04
    • Kubernetes Version : no
    question component:vineyard 
    opened by Sanzo00 12
  • [BUG] Graphscope installation fails on Apple M1/Silicon

    [BUG] Graphscope installation fails on Apple M1/Silicon

    Describe the bug When running import graphscope after installing graphscope on an Apple Silicon Mac, the Python interpreter crashes with the following error:

    [libprotobuf ERROR google/protobuf/descriptor_database.cc:559] Invalid file descriptor data passed to EncodedDescriptorDatabase::Add().
    [libprotobuf FATAL google/protobuf/descriptor.cc:1357] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
    libc++abi: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
    zsh: abort      python
    

    To Reproduce Steps to reproduce the behavior (on Apple Silicon):

    1. Install graphscope via pip install graphscope in a Python virtual environment.
    2. Run import graphscope in a Python shell inside the virtual environment.

    Expected behavior No output upon import as graphscope would have correctly initalised.

    Environment:

    • GraphScope version: 0.10.0
    • OS: macOS Monterey
    • Version: 12.3.1
    • Python version: 3.9.11
    • Kubernetes Version [e.g., 1.19]: None
    bug component:python 
    opened by gillesmag 12
  • etcd: Increate the ETCD pods period seconds

    etcd: Increate the ETCD pods period seconds

    What do these changes do?

    I just increased the ETCD pods default livenessProbe.periodSeconds from 2 to 10, with the previous config periodSeconds * failureThreshold = 2 * 8 = 16 is a bit small~

    opened by haoxins 12
  • extended k8s configuration capabilities for nodes and the coordinator

    extended k8s configuration capabilities for nodes and the coordinator

    Is your feature request related to a problem? Please describe. Currently pod configurations are created by a script. So we do not have control over these configurations. This is making it difficult to customize the graphscope deployment. For example, adding a customized mount point, setting ephemeral storage, setting CPU limits to guide the scheduler, or assigning pods to nodes using labels is not possible.

    Describe the solution you'd like Being able to configure the pods however we like.

    Describe alternatives you've considered Currently we are setting k8s defaults for some of the critical issues (ephemeral storage). But it's not a valid solution for all of the problems.

    Additional context

    enhancement kubernetes 
    opened by xarion 11
  • Can't load graph from HDFS

    Can't load graph from HDFS

    Hey GS, I try to test GraphScope with data in HDFS with the following code:

    import graphscope
    import graphscope.nx as nx
    from graphscope.framework.loader import Loader
    
    graphscope.set_option(show_log=True)
    
    edges = Loader("hdfs:///data/graph/twitter.e", host='localhost', port='8020')
    graph = graphscope.g(directed=False)
    
    graph.add_edges(edges)
    

    However I got this error:

    E0601 08:12:02.784389411  227251 fork_posix.cc:76]           Other threads are currently calling into gRPC, skipping fork() handlers
    Exception ignored in: <function Launcher.__del__ at 0x7f0f10f4a550>
    Traceback (most recent call last):
      File "/opt/app_graphyen/venv3.8/lib/python3.8/site-packages/vineyard/launcher/launcher.py", line 63, in __del__
        self.join()
      File "/opt/app_graphyen/venv3.8/lib/python3.8/site-packages/vineyard/drivers/io/stream.py", line 139, in join
        raise RuntimeError(
    RuntimeError: Subprocesses failed with the following error: 
    Failed to launch job [/opt/app_graphyen/venv3.8/lib/python3.8/site-packages/vineyard/drivers/io/ssh.sh localhost vineyard_read_bytes /tmp/vineyard.sock.1654063891.489883 "hdfs:///user/feng/notebooks/data/graph/twitter.e" eyJob3N0IjogImhkcC1saXZlLWEiLCAicG9ydCI6ICI4MDIwIn0= eyJkZWxpbWl0ZXIiOiAiLCIsICJoZWFkZXJfcm93IjogdHJ1ZSwgInNjaGVtYSI6ICIwLDEiLCAiY29sdW1uX3R5cGVzIjogImludDY0X3QsaW50NjRfdCIsICJpbmNsdWRlX2FsbF9jb2x1bW5zIjogdHJ1ZX0= 1 0], exited with 127: bash: vineyard_read_bytes: command not found
    
    extra diagnostics are as follows: 
    
    /tmp/arrow-apache-arrow-1.0.1/cpp/src/arrow/result.cc:28: ValueOrDie called on an error: NotImplemented: Got HDFS URI but Arrow compiled without HDFS support
    

    Do you know how to resolve this issue: Got HDFS URI but Arrow compiled without HDFS support? Thanks!

    bug component:vineyard 
    opened by parkerzf 10
  • string indices must be intergers

    string indices must be intergers

    sub_graph = interactive.subgraph("g.V().has('year', inside(2014, 2020)).outE('cites')") 当我运行上行代码时出现错误‘’string indices must be intergers ‘’ 请问如何解决

    WechatIMG38

    • [ ] #
    bug component:gie component:python 
    opened by banbsyip 10
  • [BUG] Latest main branch cannot pass unittest, missing few python files

    [BUG] Latest main branch cannot pass unittest, missing few python files

    Describe the bug Latest main branch cannot pass unittest, missing few python files

    To Reproduce Steps to reproduce the behavior:

    1. make test

    Screenshots If applicable, add screenshots to help explain your problem. ImportError while loading conftest '/home/graphscope/GraphScope/python/graphscope/tests/conftest.py'. graphscope/__init__.py:19: in <module> from graphscope.analytical.app import * graphscope/analytical/app/__init__.py:21: in <module> from graphscope.analytical.app.attribute_assortativity import \ graphscope/analytical/app/attribute_assortativity.py:22: in <module> from graphscope.framework.app import AppAssets graphscope/framework/app.py:30: in <module> from graphscope.framework.context import create_context_node graphscope/framework/context.py:28: in <module> from graphscope.framework import dag_utils graphscope/framework/dag_utils.py:22: in <module> from graphscope.framework import utils graphscope/framework/utils.py:39: in <module> from graphscope.framework.errors import check_argument graphscope/framework/errors.py:21: in <module> from graphscope.proto import error_codes_pb2 E ImportError: cannot import name 'error_codes_pb2' from 'graphscope.proto' (/home/graphscope/GraphScope/python/graphscope/proto/__init__.py) make: *** [unittest] Error 4

    Environment (please complete the following information):

    • GraphScope version: master
    • OS: Linux
    • Version Ubuntu 20.04
    opened by hh2712 9
  • Bump up python dependencies to latest versions

    Bump up python dependencies to latest versions

    We have several python packages has been fixed to an old version, to avoid compatibility issues. We may need consider making some changes to our codebase to drop those version limitations. And we should regularly do this version upgrading in the future. To name a few, we have those version fixed: grpcio, protobuf, networkx, pandas, pyarrow

    enhancement component:python 
    opened by siyuan0322 0
  • Make wheels support Python3.10, 3.11

    Make wheels support Python3.10, 3.11

    Is your feature request related to a problem? Please describe.

    The default apt repository of ubuntu:latest has a minimum Python version of 3.10, I think we should make our wheels support 3.10 and 3.11 as well.

    component:python 
    opened by siyuan0322 0
  • [BUG] GIE tests become flaky on CI

    [BUG] GIE tests become flaky on CI

    Describe the bug

    Looks like recently PRs on GIE introduce some flaky tests on CI:

    • https://github.com/alibaba/GraphScope/actions/runs/3562015896/jobs/5983399091
    • https://github.com/alibaba/GraphScope/actions/runs/3562011371/jobs/5983853239

    Both failure happened in GIE and that should be locally reproducible, I think.

    Additional context Add any other context about the problem here.

    component:gie component:dev-infra 
    opened by sighingnow 0
  • Algorithms in pregel written by myself is inefficient

    Algorithms in pregel written by myself is inefficient

    The code in the sample takes 15s, which is 10 times as long as the built-in algorithm(1.3s).

    • OS: Ubuntu 18.04.6
    • Memory: 128G
    • Core: 16
    • Python: 3.8.6
    • GraphScope: 0.17.0
    • Graph File : LiveJournal social network
    • cluster_type: hosts
    • num_workers: 1
    component:gae priority:high performance 
    opened by GongChangYan 1
Releases(v0.18.0)
  • v0.18.0(Dec 2, 2022)

    We are delighted to bring a number of improvements to GraphScope, alongside the GraphScope 0.18.0 release. This release contains many important features and enhancements to Graph Interactive Engine (GIE), including introducing a new strategy for pattern-matching queries and supporting canceling running queries. The Graph Learning Engine (GLE) now supports PyTorch and is also compatible with PyG. In addition, we take a first step towards modularized deployment for different components of GraphScope in Kubernetes.

    We highlight the following improvements included in this release:

    1. Enhancements for GIE

    • A new execution strategy based on worst-case optimal join is introduced to GIE engine, potentially improving the performance of match step by orders of magnitude.
    • The query can be canceled after its execution exceeds a pre-given overtime parameter (10min by default).
    • GIE engine supports a failover mechanism: if an executor pod fails, but the data is not missing, it can be restarted by k8s. Existing queries cannot recover, but the engine can serve the following.

    2. Enhancements for GAE

    • Add more variants of WCC algorithm.
    • Supports local vertex map to make it could scale to larger graphs given more workers.

    3. Enhancements for GLE

    • Add support for PyTorch and PyG
    • Add heterogeneous graph support for subgraph-based GNN, add HeteroSubGraph and HeteroConv, bipartite GraphSAGE and UltraGCN.
    • Add edge feature support in both EgoGraph and SubGraph.
    • Add recommendation metrics: Recall, NDCG and HitRate.
    • Add hiactor-based graph engine.

    4. Standalone deployment

    • Supports a version of GIE standalone deployment in Kubernetes.
    • Redesigned docker files from the bottom to the top, making it more clear and more concise.

    5. GAE Java Enhancement

    • Introduce @GrapeSkip for better user experience in template method overloading.
    • Speedup Java App runimte codegen.

    Docker Image

    docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.18.0
    
    # GraphScope persistent storage, user can only perform GIE query on it.
    docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope-store:0.18.0
    

    Commits

    • [6775658d7]: Push jupyter image nightly with the latest client (#2266) (DongZe Li)
    • [df9f6ebf3]: Fix nightly CI (#2262) (Siyuan Zhang)
    • [4f1777700]: Fixes CI failure both on Linux and MacOS (#2244) (DongZe Li)
    • [07efc740d]: [GIE/engine] Bug fix; (#2250) (bmmcq)
    • [5ea5b874a]: [GIE] Support parallel scan on ExpStore (#2253) (BingqingLyu)
    • [432e65c89]: [GIE] Make the version of GIE compiler consistent with the default value in interactive engine pom (#2249) (shirly121)
    • [2b7cf0050]: [GIE POM] Unify GIE version by the global flag 'revision' (#2243) (shirly121)
    • [71d61e3cf]: Reorganize for Makefiles and dockerfiles (#2246) (Siyuan Zhang)
    • [84f449c7c]: [GIE Runtime] Debug runtime op info on in a more clearly way (#2226) (BingqingLyu)
    • [f77627dd7]: Update the learning model to align with the latest graphlearn (#2235) (Tao He)
    • [1c4c1cd17]: [GIE/Store] Reduce memory footprint in exp_store (#2245) (Longbin Lai)
    • [e375f8d3d]: [GIE/Exp-Store] Optimize memory usage in loading graph data into MutTopo. (#2242) (Longbin Lai)
    • [8844e9027]: Fixes release graphscope-store image (#2241) (Siyuan Zhang)
    • [428c7855c]: [GIE/Engine]: add direct executor implementation; (#2234) (bmmcq)
    • [e071ce4eb]: add docs revision preview in CI (#2237) (Jingbo Xu)
    • [f52455078]: Enable cpplint in CI (#2233) (Tao He)
    • [330fe14a7]: Makes GAE compatible with latest arrow release (#2232) (Tao He)
    • [ed8604036]: Revise docs. (#2231) (Jingbo Xu)
    • [b4d7038da]: Fixes compilation error of graphx runner (#2228) (Siyuan Zhang)
    • [a11b45395]: Reorganize Dockerfile according to specification of standalone deployment (#2227) (Siyuan Zhang)
    • [dcaa38ac0]: [GIE/GLogue] rename Catalogue as GLogue (#2224) (BingqingLyu)
    • [aed1f1362]: [GIE/IR] Introduce a new strategy for pattern matching (#2159) (BingqingLyu)
    • [92f7da3a9]: Add two other version of wcc for compatiblity (#2223) (Siyuan Zhang)
    • [2fbc4ad7c]: Add local vertex map to arrow fragment and arrow projected fragment. (#2212) (Siyuan Zhang)
    • [cc5170434]: Fixes nightly CI on macOS (#2219) (DongZe Li)
    • [f21754fcc]: [GAE-Java] GRAPE JDK support Spark local mode for graphx (#2216) (Zhang Lei)
    • [ed33b08c6]: [BugFix] GAE JavaDoc not generated (#2206) (Zhang Lei)
    • [440572158]: Update the DefaultTag to .Chart.AppVersion (#2210) (Ye Cao)
    • [3ee05f77b]: [new FAQ] add network proxy related FAQ (#2211) (Yifan Yuan)
    • [09995bfa0]: Delegate parquet/orc files to vineyard io adaptors (#2209) (Tao He)
    • [7ecd75a47]: Delete all deployed resources without a extra rbac role when using helm unintsall (#2208) (Ye Cao)
    • [094e81d67]: [Bug Fix] Add default tag (.Chart.AppVersion) to graphscope-store (#2205) (Ye Cao)
    • [4279b8fc8]: [BugFix] make ResultCode in ir core ffi consistent with Compiler (#2188) (BingqingLyu)
    • [7cfe9a3fd]: [Bug Fix] Avoid throwing exception after processing grpc error from executor in compiler (#2182) (shirly121)
    • [abdd5c979]: [GAE-Java] Update doc (#2199) (Zhang Lei)
    • [8de000fd1]: [GAE-Java] Remove String-specialized FFI classes and Simplify Java MessageManager interface (#2195) (Zhang Lei)
    • [5a4312ded]: Fix failure in network forward CI (#2204) (Siyuan Zhang)
    • [773545142]: Fixes failures in nightly CI (#2201) (Tao He)
    • [264cbc8a2]: Log level can accept int and string ignore case (#2200) (Siyuan Zhang)
    • [fb12c7b12]: Fix slf4j version conflicts (#2196) (Siyuan Zhang)
    • [87b960232]: [GIE Deploy] Deploy GIE with exp or v6d in standalone mode by helm configuration management (#2143) (shirly121)
    • [ecd58a248]: Fix possible java sandbox protection. (#2191) (Siyuan Zhang)
    • [38a3781dc]: Enable HDFS tests in CI. (#2175) (Tao He)
    • [7fdcd2357]: [GAE-Java] Support grape-graphx running on yarn cluster and support multiple executor on one host (#2185) (Zhang Lei)
    • [f983bd672]: [GIE pegasus] Support cancelling query. (#2168) (Neng Li)
    • [26f03827e]: gss ci use lesser image size (#2186) (Siyuan Zhang)
    • [2b00169b9]: [GAE-Java] Fix GAE-Java driver app's message strategy specification (#2178) (Zhang Lei)
    • [f4dea3e74]: [OSPP] Spark RDD Reader for GraphScope (#2103) (Issac-Newton)
    • [cfb707dd7]: Enhance grpc finding in GAE cmake (#2184) (Tao He)
    • [25ab4664f]: Use pre-built llvm instead of build when making analytical docker (#2183) (Siyuan Zhang)
    • [79a08d3b9]: [GIE Compiler] update ResultCode to be consistent with ir core interface (#2158) (shirly121)
    • [74af74f00]: [GAE-Java] GraphX-on-GraphScope : remove GraphXFragment and related components (#2123) (Zhang Lei)
    • [49ae814f1]: [GAE-Java] Add necessary configurations in pom.xml to publish GRAPE jdk to maven central repo (#2166) (Zhang Lei)
    • [2821169a8]: Fix create interactive engine failure on helm installed graphscope (#2171) (Siyuan Zhang)
    • [89a4a218e]: [GAE-Java] GRAPE jdk submodules naming to same pattern (#2163) (Zhang Lei)
    • [321d93805]: Upgrade GitHub actions versions (#2160) (Siyuan Zhang)
    • [695cb9caa]: Reorganized coordinator and redesigned the logic to create engines. (#2152) (Siyuan Zhang)
    • [ea4ca1d0b]: fix src label and dst label of edge kinds may be empty (#2153) (Siyuan Zhang)
    • [1b312af5c]: Move the tolerances of graphscope-store charts to the global scope (#2151) (DongZe Li)
    • [37fa85ad4]: [GAE-Java] Speed up Java App runtime codegen& compiling (#2150) (Zhang Lei)
    • [733fda3d0]: [GIE IR] Add docker files to build images of GIE (using exp store) (#2142) (shirly121)
    • [0c5a4b55d]: [GAE-Java] Fix : can not compile the submodule of GAE Java SDK (#2146) (Zhang Lei)
    • [3220a7f03]: Fixes the conditions for GAE CI in PR and on main, add timeout for test_java_app.py (#2148) (Tao He)
    • [998e3615c]: Rename pegasus.hosts to network.servers for executor (#2147) (Siyuan Zhang)
    • [97eec7023]: Add pytest-timeout to requirements-dev.txt and fixes a deprecated warnings when running tests (#2145) (Tao He)
    • [3038b8970]: [GAE-Java] Introduce @GrapeSkip to support generated method overloading (#2144) (Zhang Lei)
    • [77b4ec5c6]: [GAE-Java] Fix log4j jar missing in grape-runtime and java pie driver app's message strategy settable. (#2140) (Zhang Lei)
    • [7d351a2b5]: [IR Compiler] make gremlin server listen to 0.0.0.0 instead of localhost (#2141) (shirly121)
    • [759b7a5b0]: Server now use aks provided by client side (#2132) (Siyuan Zhang)
    • [5c963f583]: Update cluster.py to fix the bug that etcd selector could not work (#2139) (changyi1223)
    • [6d94c3fa0]: [Bug Fix] Fix bug in index_scan on Groot/Vineyard (#2129) (BingqingLyu)
    • [d4d942c51]: [Assembly] wait_servers_ready() instead of sleep 3s when start engine on vineyard (#2136) (BingqingLyu)
    • [2aa3cfc38]: Update commons-codec:commons-codec 1.11 to 1.13 (#2119) (zhoumengyks)
    • [6d3d6aada]: Update org.apache.kafka:kafka-clients 2.5.0 to 2.7.2 (#2120) (zhoumengyks)
    • [6d5d288fa]: Fixes the docstring rendering for GAE apps (#2131) (Tao He)
    • [950ed7ab7]: App now can carry cmake options, and enabled wcc on string oid. (#2130) (Siyuan Zhang)
    • [31829fd08]: [GIE/Runtime] Refine some error infos in Runtime (#2113) (BingqingLyu)
    • [d16b1b482]: Fixes configuration in dockerignore (#2126) (Tao He)
    • [71c37fecf]: Upgrade com.fasterxml.jackson.core:jackson-databind to version 2.14.0-rc1 (#2122) (Siyuan Zhang)
    • [ace93c53b]: [Pegasus] minor fix: avoid iter.next() when it is None in release (#2112) (BingqingLyu)
    • [5756de3dc]: Make gae CI invoked when CMakeLists.template changes (#2101) (Zhang Lei)
    • [b6d68bc61]: New predicate support (#2086) (waruto)
    • [eb393dab1]: Bump protobuf-java from 3.18.2 to 3.19.6 in /interactive_engine (#2109) (dependabot[bot])
    • [0b10f5c0f]: Bump protobuf-java from 3.18.2 to 3.19.6 in /interactive_engine/compiler (#2111) (dependabot[bot])
    • [b1132cc7c]: [ASoC] report for subtask (#2107) (Haochen Shi)
    • [751e3e7c6]: [GIE IR] Provide expr store ffi interfaces to encode and write data (#2098) (shirly121)
    • [91db0334b]: [IR Runtime] Support removing columns in Record via Auxilia in IR-Runtime (#2085) (BingqingLyu)
    • [e024c6549]: [GIE-IR] Fix the doc inconsistency issue of the legacy MaxGraph and the latest Gaia-IR. (#2074) (shirly121)
    • [cf7495c69]: Pre-release v0.18.0 (#2104) (DongZe Li)
    • [5aabc4dc0]: [IR Benchmark] Benchmark for BI queries on GIE-IR (#2081) (BingqingLyu)
    • [be1d7a864]: upgrade rocksdb-rust to 0.19.0 (#2100) (Siyuan Zhang)

    New Contributors

    • @Swthaochen made their first contribution in https://github.com/alibaba/GraphScope/pull/2107
    • @zhoumengyks made their first contribution in https://github.com/alibaba/GraphScope/pull/2120
    • @changyi1223 made their first contribution in https://github.com/alibaba/GraphScope/pull/2139
    • @Issac-Newton made their first contribution in https://github.com/alibaba/GraphScope/pull/2103
    • @YifanYuan3 made their first contribution in https://github.com/alibaba/GraphScope/pull/2211

    Full Changelog: https://github.com/alibaba/GraphScope/compare/v0.17.0...v0.18.0

    Source code(tar.gz)
    Source code(zip)
  • v0.17.0(Sep 29, 2022)

    We are glad to announce a number of new features and improvements to GraphScope, alongside the GraphScope 0.17 release. The updates include new releases for Spark-GraphX support, backend engine enhancement, and frontend APIs. Currently, GraphX Pregel algorithms can be seamlessly executed on GraphScope. A lot of new features have also been officially brought into the interactive engine (GIE), including the syntactic sugar of path expansion and expression, the Gremlin steps of id(), label(), coin() and otherV() etc. and the profiling and benchmarking tools for LDBC BI queries.

    We highlight the following improvements included in this release:

    1. Spark-GraphX support for GAE:

    • Support converting GraphX graph to GraphScope Fragment
    • Support wrapping Fragment as GraphX Graph.
    • Support running GraphX pregel on GraphScope Analytical engine.

    2. New language features and enhancement for GIE:

    • The syntactic sugar of path expansion. Now user can write out[in|both]('x..y', 'KNOWS') to support path expansion of at least x (included) times and at most y (excluded) times from a certain node. with() step can be followed to configure the path expansion.
    • The syntactic sugar of expression, introduced by expr(), can be used along with the where()-step to enable filtering expression. For example, .where(expr("@.name == \"John\"")) help filter out the current entry that has the name of "John". We have updated the README doc for some instructions.
    • Provide LDBC benchmarking for BI queries.

    3. Enhancement and dependency refinement of Groot to reduce the size of fat-jar.

    4. Feature updates:

    • Add from_dict / to_dict option for graph schema.
    • Add syntactic sugar of path expansion, expr(), and the Gremlin steps of id(), label(), coin(), TextP.startsWith/endsWith... etc.

    Docker Image

    docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.17.0
    docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope-store:0.17.0
    

    Commits

    • [e0a761580]: Get the exception type in correct way, both on Linux and on MacOS. (#2099) (Tao He)
    • [0fcc2d815]: Update pom.xml (#2095) (Rudolf D)
    • [2caa69b7d]: Fixes the missing return statement in frames (#2097) (Tao He)
    • [12f32fff4]: Ensure the correctness of retry (in the RunStep request) (#2094) (Tao He)
    • [f3295f4a5]: Test against the latest libgrape-lite and vineyard (#2090) (Tao He)
    • [4caa4b8ed]: Enhance the error handling in frames (#2089) (Tao He)
    • [ea7a18952]: Fixes the typo in error message template (#2092) (Tao He)
    • [a69d884f3]: Closing the session if hearbeat fails for many times (when engine crashes) (#2088) (Tao He)
    • [2119d4851]: Add nodeselector arguments to control where the pod will be launched (#2087) (Tao He)
    • [9f1f64410]: Change nightly image tag to 'major.minor.alpha_patch' (#2075) (DongZe Li)
    • [1846e901a]: [GAE-Java] Make GAE Java SDK version align with GraphScope (#2077) (Zhang Lei)
    • [652b1c5d6]: [GAE-Java] Upgrade log4j version (#2082) (Zhang Lei)
    • [28809ddef]: [GIE-IR] Profiling for GIE-IR (#2056) (BingqingLyu)
    • [2d57a3d8b]: [GIE-IR] support sending back a page of results if batch size met (#2076) (shirly121)
    • [a7462ae13]: Fixes GetMessage bug in GAE java SDK (#2079) (Zhang Lei)
    • [c4572e69d]: [GIE-IR] support more patterns of predicate, i.e. P.not, P.inside, P.outside (#2068) (shirly121)
    • [c463b3287]: Enhancement and dependency clearing of groot. (#2043) (Siyuan Zhang)
    • [3af3e84e9]: [IR Compiler] fix expand fusion strategy (#2058) (shirly121)
    • [45e8c0503]: Change builtin apps(bfs,sssp,pagerank) to parallel version (#2073) (DongZe Li)
    • [56a3b9c14]: Update GAE Java Doc (#2059) (Zhang Lei)
    • [a7ff7b666]: [IR Runtime] fix bug in label encoding of exp_store (#2057) (BingqingLyu)
    • [1ddf147bf]: [IR Compiler] support more project patterns [id(), label(), constant(XX)] (#2066) (shirly121)
    • [5f4813fd4]: [Emergency Fix]Compile Property Graph with ENABLE_JAVA_SDK on needs jni.h (#2069) (Zhang Lei)
    • [94a013155]: Fix GAE-Bug: Cannot compile GAE app if java not installed (#2060) (Zhang Lei)
    • [7741be002]: test filter push down (#2003) (waruto)
    • [588ce2969]: [Pre-commit Hook] Update README.md (#2064) (shirly121)
    • [516b24aa8]: [Pre-commit Hook] add pre-commit hook to prevent committing sensitive information (#2055) (shirly121)
    • [2f30b2355]: Install graphx-runner only whe it is built (#2062) (Tao He)
    • [4c7be5f33]: Fixes CI on macOS (#2061) (DongZe Li)
    • [c35766a44]: GraphScope for Spark-GraphX (#2004) (Zhang Lei)
    • [96d3bb1ff]: Fixes an uninitialized field bug in FFI reader (#2054) (Tao He)
    • [0b7483f84]: Support statements like vd_type='str' in UDF app (#2052) (Siyuan Zhang)
    • [adbd4d1d4]: Add prometheus exporter for coordinator && GIE engine (#1947) (Vincent)
    • [e87e99776]: Fixes the GAE jar name when we have a alpha in version number (#2050) (Tao He)
    • [8bf6afe32]: [GAE-Java] Code refactor (#2049) (Zhang Lei)
    • [248c13b43]: The ExceptionUtils.getRootCause() may returns null, record the error and backtrace in such cases (#2048) (Tao He)
    • [cce30bb45]: Bump vineyard to 0.8.5 (#2047) (Zhang Lei)
    • [67a45aad5]: Tensorflow requires lower version of pandas (#2045) (Tao He)
    • [486283517]: Upgrade required vineyard to v0.8.4. (#2042) (Tao He)
    • [a2e0c83f6]: [GIE/IR] Support otherV() on Groot/Vineyard (#2034) (BingqingLyu)
    • [819ea4a96]: remove some unused files in sdk-common (#2026) (Siyuan Zhang)
    • [958c49dd2]: Avoid install java related targets if java sdk is not enabled. (#2037) (Tao He)
    • [2316f0731]: Fixes the path of gaia executor, add a dockerignore file as well (#2039) (Tao He)
    • [9cdd26f41]: Fix incorrect argc value in run_vy_app.cc (#2035) (Siyuan Zhang)
    • [dc0662ecc]: upgrade vineyard to 0.8.2 (#2025) (Zhang Lei)
    • [73c1882f5]: GAE java build refactor (#2028) (Zhang Lei)
    • [e7a6655f3]: [pegasus] update contains_source in progress (#2008) (Neng Li)
    • [dd4254473]: fix bug in extract_needed_columns (#2015) (waruto)
    • [159a27b24]: [GIE-IR] support SimplePath and AllV in path expand operator (#2005) (shirly121)
    • [1f9802188]: Add connected() in session.py to figure out session status (#2012) (dzhiwei)
    • [a2f56ce8f]: Ensure session close work when triggered not in main thread (#2009) (dzhiwei)
    • [71e0bf2cf]: Fixes a typo in the tutorial notebooks (#2007) (Tao He)
    • [6dff1dc29]: Use a getaddrinfo resolvable hostname as the local hostname (will be used as the etcd endpoint further). (#2006) (Tao He)
    • [be0708793]: Return primaryKey for vertex/edge property (#2001) (DongZe Li)
    • [7f389be2d]: remove useless code (#2002) (waruto)
    • [2d6868019]: impl new condition row filter and column filter (#1905) (waruto)
    • [2ac4f24af]: Fixes a incorrect collect type in ir (#2000) (waruto)
    • [8ab15d19b]: [GIE-Runtime] Refine structure Record in Runtime (#1992) (BingqingLyu)
    • [73b54533f]: [GIE-IR] support coin(..) for sampling vertices/edges (#1993) (shirly121)
    • [c8bf5b1e1]: No fatal even when meet unknown errors (#1991) (Tao He)
    • [ce4f0eace]: [GIE] fuse expand with count operators nested in apply for optimization (#1987) (shirly121)
    • [bd8252fc3]: Drop from vineyard when unloading a arrow projected fragment (#1984) (Tao He)
    • [23dcae0fb]: Add deployment of groot (#1858) (Siyuan Zhang)
    • [4aac5b579]: [IR Compiler] collect metrics from compiler to report qps/rt (#1978) (shirly121)
    • [7b20ab507]: [IR Compiler][Bug Fix] fix gaia-test ci caused by invalid group type used in compiler (#1979) (shirly121)
    • [48d3c315a]: [GAIA-IR] support patterns of group by multiple keys/values in compiler (#1973) (shirly121)
    • [44597c93c]: [Bug Fix] fix a bug in encode/decode in group (#1966) (BingqingLyu)
    • [7d8f8cfdf]: Fix network connection reconnect abnormal in pegasus (#1969) (bmmcq)
    • [c980ab37a]: Fixes the compliation error of run_property_ctx and addresses some lint errors (#1974) (Tao He)
    • [15fdf2f39]: Fix timeout error in graphscope-store helm test (#1972) (Siyuan Zhang)
    • [de490cd67]: Move compiler into interactive_engine (#1929) (Siyuan Zhang)
    • [2f7d1ec59]: Bump hadoop-common from 3.2.3 to 3.2.4 in /interactive_engine (#1963) (dependabot[bot])

    New Contributors

    • @bifenglin made their first contribution in https://github.com/alibaba/GraphScope/pull/1931
    • @dzhiwei made their first contribution in https://github.com/alibaba/GraphScope/pull/2009
    • @VincentFF made their first contribution in https://github.com/alibaba/GraphScope/pull/1947
    • @Disseminator made their first contribution in https://github.com/alibaba/GraphScope/pull/2095

    Full Changelog: https://github.com/alibaba/GraphScope/compare/v0.16.0...v0.17.0

    Source code(tar.gz)
    Source code(zip)
  • v0.16.0(Aug 12, 2022)

    We are bringing a number of improvements to GraphScope, alongside the GraphScope 0.16.0 release. This release introduces many new features on backend engines and system stability. We enable Jave SDK of Graph Analytics Engine (GAE) on MacOS, and make a series of enhancements on the graph storage Groot. We completely remove a legacy Graph Interactive Engine (GIE), while officially replacing it with the latest version based on an intermediate representation (IR) layer; in addition, we improve GIE via a couple of optimizations, new language features, and bug fixes. The Graph Learning Engine (GLE) now supports real-time sampling on dynamic graphs. Meanwhile, we start to release a nightly version every day, and you can try it with pip3 install graphscope --pre.

    We highlight the following improvements included in this release:

    1. Enhancements of the graph storage Groot:

    • Support load table from partitioned ODPS table;
    • Speed up some popular gremlin queries;
    • Use the GAIA engine to support queries;
    • A series of deployment enhancements.

    2. Enhancements of the GIE GAIA engine:

    • Completely replacing the legacy GIE engine with the latest version based on IR;
    • Optimizing the logic of lazily fetching properties from the graph store, and implementing a new CSR-based storage for experiment purpose;
    • New language features including more grouping functions, more options for group/dedup/orderby keys, etc.

    3. Dynamic-Graph-Service (DGS) is introduced in GLE.

    • DGS is an online inference service, it supports real-time sampling on dynamic graphs with streaming graph updates.

    4. Support Java SDK of GAE for MacOS.

    5. Bug fixes:

    • Fixes potential pointer leak in IR-core FFI functions;
    • Duplicate results while scanning with primary keys;
    • Error while accessing a tag that does not exist in the record.
    • Fixes failure of running clustering algorithm on the property graph.

    Docker Image

    docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.16.0
    

    Commits

    • [9c656fa52]: [GAIA Runtime] Use hashbrown for optimization (#1952) (BingqingLyu)
    • [a27dd5c12]: Support for version suffixes like '0.15.0a1' in cmake (#1960) (DongZe Li)
    • [e6917769b]: upgrade protoc-gen-grpc-java version (#1830) (Siyuan Zhang)
    • [9a71bf4c9]: Fixed high CPU usage of ingestor when having empty buffer (#1946) (simple)
    • [3a2165da1]: fix: add ipc sender after former one removed (#1950) (Neng Li)
    • [e486c35e3]: [GAIA/exp_store] To enable getting neighbors of a certain label by la… (#1945) (Longbin Lai)
    • [7f0f05398]: [pegasus] fix bug in (#1954) (#1958) (bmmcq)
    • [009fd8c32]: Fixes nightly CI by overwrite VERSION (#1957) (DongZe Li)
    • [f6dcf166d]: Change nightly image tag to 'nightly' (#1951) (DongZe Li)
    • [84e8de89d]: Support to pre-release GraphScope(image + wheel package) nightly with version like 'major.minor.alpha_patch' (#1949) (DongZe Li)
    • [5d9b62938]: Fix typos in the tutorials (#1948) (Ye Cao)
    • [1de753389]: Fix missing dependency of ir-compiler (#1935) (Siyuan Zhang)
    • [a0da2a974]: [GAIA-IR] create ffi object from java side to avoid invoking ffi interface which can lead memory leak problems (#1925) (shirly121)
    • [9284f69c5]: [GraphProxy] Optimize LazyDetails (#1921) (BingqingLyu)
    • [98ae038f5]: Fixes the links for tutorials in the documentation. (#1928) (Tao He)
    • [52078d9a4]: Integrate targets of assembly artifacts in GIE (#1920) (Siyuan Zhang)
    • [5e93f732e]: [IR Compiler] fix bugs when path_expand is nested in sub traversal, i.e. select('a').by(out('1..2')) (#1874) (shirly121)
    • [281f264f9]: Support edges of outer vertices in DynamicFragment (#1799) (ds-ssj)
    • [489a07725]: Fixed graph schema drop and init error when edge_kinds's src or dst gone (#1922) (simple)
    • [aedbb31f3]: Fix compiling error of write-bench (#1918) (Siyuan Zhang)
    • [7009ccaa9]: [IR Compiler] create FfiNameOrId from java side to fix undefined error (#1919) (shirly121)
    • [7e826c67f]: Cleanup legacy files. (#1914) (Siyuan Zhang)
    • [8a04f8384]: Print the running time of each round of analytical engine (#1913) (DongZe Li)
    • [41fe90e41]: Add the etcd {client,peer} port to the configuration list (#1910) (Tao He)
    • [484f9278d]: Reopen groot gremlin test (#1912) (Siyuan Zhang)
    • [160bb0b08]: Resolve issues reported by dependent bot (#1909) (Siyuan Zhang)
    • [1591300bf]: impl partial_eq for Property and fix gitignore (#1908) (waruto)
    • [eb45c584c]: [GIE] Reorganize executor package by incorporating GAIA into GIE (#1883) (BingqingLyu)
    • [289990309]: Data load tools supports partition (#1900) (Siyuan Zhang)
    • [cd08a1a47]: Enhance the install_deps.sh script (#1873) (Weibin Zeng)
    • [a33b179a4]: [GAIA-IR] support more patterns of dedup().by() in ir compiler (#1901) (shirly121)
    • [7087915d7]: Supporting reading from a config file in htap loader. (#1897) (Tao He)
    • [b0c9931a0]: Fix a bug when iterating dict (#1894) (Siyuan Zhang)
    • [2ac521374]: [GAIA-IR] support more patterns of aggregate functions, i.e min/max/toList/toSet/avg/countDistinct (#1847) (shirly121)
    • [28508f6f8]: Fix docs words error (#1893) (Milittle)
    • [13aca2ec5]: add arm runtime dockerfile (#1880) (Siyuan Zhang)
    • [1e35af2b5]: Reopen releasing gss in workflow (#1882) (Siyuan Zhang)
    • [b22e45e4a]: [GAIA-IR] Support PathExpand ranging from 0 (#1863) (BingqingLyu)
    • [e96e6ec78]: Fixes the dtor error in the htap_loader. (#1884) (Tao He)
    • [7770814e5]: Adapt the launch cluster script to latest AWS API (#1876) (Weibin Zeng)
    • [9e1a3346c]: Add a standalone graph loader to vineyard. (#1869) (Tao He)
    • [e5cdf481d]: Avoid mvn install fastffi twice (#1871) (Zhang Lei)
    • [51296d129]: Decrease the make job numbers when building gae. (#1868) (Siyuan Zhang)
    • [ea825d69a]: Optimize the clustering algorithm by replacing the complex vertex_array_t with unordered_map (#1861) (Weibin Zeng)
    • [d024cdab3]: [Bug Fix] fix a bug in duplicated results by index scan (#1865) (BingqingLyu)
    • [fd9b55f90]: [GAIA] Make input data as generic data types in JobAssembly interface (#1859) (BingqingLyu)
    • [6616ee08a]: [IR Runtime] support mean() in runtime (#1852) (BingqingLyu)
    • [0a8e9c1d8]: [IR Compiler] support hasNot in ir compiler (#1853) (shirly121)
    • [a30384cfe]: [GAIA-IR] Filter the data when selecting a non-exist tag (#1843) (BingqingLyu)
    • [269578393]: fix groot and GIE test cases after replacing MaxGraph with gaia (#1823) (shirly121)
    • [959f6ff08]: [GAIA-IR] Fix invalid unwrap in ffi functions to prevent broken pointers (#1846) (shirly121)
    • [ffeebcd87]: [GIE/TEST] Add back CI Test for subgraph() (#1827) (BingqingLyu)
    • [a756bc1ca]: [GIE-GAIA] Add Pegasus named apis, and add names for IR Operators (#1834) (BingqingLyu)
    • [5698d98ba]: Add a TOC to the README. (#1838) (Tao He)
    • [1d9222738]: Fixes the long standing empty column name issue in subgraph (#1832) (Tao He)
    • [86ea24bcd]: Upgrade protoc version and use variable to replace hardcoded value (#1829) (Siyuan Zhang)
    • [2eb27008c]: Remove Zookeeper and zetcd related stuffs (#1828) (Siyuan Zhang)
    • [c35f34181]: [pegasus] fix defect in the progress control (#1796) (Neng Li)
    • [35b56f2d6]: Make Pegasus support reading config files with hostname instead of only IP (#1808) (Melo Yang)
    • [2fe38e8c9]: Update dependency versions. Add rust format check (#1821) (Siyuan Zhang)
    • [7373a7f25]: Add GRAPE-jdk support on mac (#1774) (Zhang Lei)
    • [36820c4f1]: [GAIA-IR] Support sum() in GAIA-Runtime (#1815) (BingqingLyu)
    • [f96094664]: [GIE/IR] Fix IR warnings (#1817) (Longbin Lai)
    • [bded1bf14]: use the same format file as query_service/ir (#1819) (waruto)
    • [a675598e5]: Workaround for pip failure to perform editable install with --user (#1813) (Weibin Zeng)
    • [d49476af5]: Introduce a new interactive engine (#1782) (Siyuan Zhang)
    • [889e00355]: Add degree threshold to katz_centrality, clustering and avg_clustering algorithm (#1507) (Siyuan Zhang)
    • [d1e507616]: [IR Runtime] Make v6d related codes as features (#1795) (BingqingLyu)
    Source code(tar.gz)
    Source code(zip)
  • v0.15.0(Jul 6, 2022)

    We highlight the following fixes and improvements included in this release:

    • [GAIA-IR] Fixes the multi-join error in the plan of pattern matching
    • Supports bulk load from ODPS table and OSS for the Interactive Engine.
    • [GAIA-IR] Support the bothV() operator.
    • Compatible with the latest GLOG API of Analytical Engine.
    • Fixes some compilation errors in Learning Engine under Apple M1 chip.
    • [GAIA-IR] fix the protobuf encode error of logical plan in Interactive Engine.
    • Fixes bug while installing dependencies on ubuntu and update deprecated --all to --workspace

    Docker Image

    docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.15.0
    

    Commits

    • [b93a80942]: Fix nightly CI on macOS (#1797) (DongZe Li)
    • [f7c18d16f]: ignore a Cargo.lock (#1792) (waruto)
    • [175852558]: Impl PartialOrd and some methods for Property, then format it (#1788) (waruto)
    • [e8f0b98b5]: [Gaia/IR] Return json with possible error message in print_plan_as_json (#1786) (Longbin Lai)
    • [5af6ef948]: Fixes bug while install deps on ubuntu and update deprecated --all to --workspace (#1785) (waruto)
    • [314a3fc2a]: [GAIA-IR] fix a bug of encode protobuf of logical plan (#1784) (shirly121)
    • [0d8fca9cc]: [GAIA-IR] Fix compiling warnings in v6d_ffi (#1780) (BingqingLyu)
    • [07e86722f]: Fixes the failure of nightly CI on macOS by fixed the rust toolchain version to 1.60.0. (#1777) (DongZe Li)
    • [e0166e70b]: fix communication caused by broadcast (#1750) (Neng Li)
    • [b7960ba0f]: [GAIA-IR] Make v6d_ffi build as features in IR (#1775) (BingqingLyu)
    • [9abea6d4a]: support more than 2 branches to union (#1756) (Melo Yang)
    • [faadf9f05]: [GAIA-IR] Support writing a new graph in GAIA-IR (#1745) (BingqingLyu)
    • [3e449f981]: [IR Compiler] support TextP.containing/TextP.notContaing as within/without in ir_core (#1755) (shirly121)
    • [ec11bd169]: [IR Compiler] represent identity as AsNone in subgraph (#1752) (shirly121)
    • [cfabc90d6]: fix bug, or_predicates is a collection of AndPredicate that forms a logical OR of all AndPredicates (#1751) (waruto)
    • [2cba97080]: [IR Compiler] fix error when creating json file (#1743) (shirly121)
    • [b3f9afe80]: [GAIA-IR] support lazy projection for columns used only in a predicate (#1746) (shirly121)
    • [1ab5737f4]: Bump fastjson from 1.2.76 to 1.2.83 in /analytical_engine/java (#1732) (dependabot[bot])
    • [be3bf0ee4]: Bump fastjson from 1.2.75 to 1.2.83 in /interactive_engine (#1731) (dependabot[bot])
    • [f8448083b]: Upgrade vineyard version to fixes some build failures on Mac. (#1748) (Tao He)
    • [96ffefee2]: [GAIA-IR] support subgraph in ir compiler (#1721) (shirly121)
    • [a97a7791a]: Upgrade required graphlearn to fixes a build error on M1 Mac. (#1742) (Tao He)
    • [7af26c96c]: Compatible with the latest API of GLOG (#1738) (DongZe Li)
    • [bcbe64609]: Fix bugs while building on macOS 12 wih apple silicon (#1736) (waruto)
    • [f25fc429f]: Publish dataset image during release process (#1730) (DongZe Li)
    • [92f6564d3]: Multi task for bulk load (#1720) (Yongmin Hu)
    • [bcd37732a]: add crc to file meta store (#1726) (tianliplus)
    • [fe9d03383]: Supports launch vineyardd using on multiple hosts using MPI (#1727) (Tao He)
    • [afd61fd75]: [GAIA-IR] support bothV() in GAIA-IR (#1717) (BingqingLyu)
    • [deb47c9d2]: integrate gremlin query api into groot sdk and add authentication when querying (#1715) (shirly121)
    • [85a70b884]: Stores download OSS files with multithreading for GIE (#1714) (Yongmin Hu)
    • [333c3bead]: [GAIA-IR] Reorg for GAIA-IR's Runtime (#1709) (BingqingLyu)
    • [ded5b2efc]: Supports bulk load from ODPS table and OSS for GIE (#1672) (Yongmin Hu)
    • [0b49a06da]: [Bug fix - Gaia-IR] The plan of pattern matching faces a multi-join error (#1704) (Longbin Lai)
    • [66a4f06bb]: Disable publish the wheel package to Test.Pypi (#1700) (DongZe Li)
    Source code(tar.gz)
    Source code(zip)
  • v0.14.0(Jun 9, 2022)

    We are delighted to present the release of GraphScope v0.14.0. This release is composed of many updates on backend engines and system stability. The new Graph Interactive Engine (GIE), GAIA-IR, has supported more operators for diverse graph queries. Meanwhile, the persistent storage of GraphScope is further enhanced with a series of new functions. In addition, we continuously work on improving the performance of GraphScope, and developer usability.

    We highlight the following improvements included in this release:

    1. New supported operators in Graph Interactive Engine (GIE)

    • Support valueMap operator to fetch all properties of vertices in the GAIA engine.
    • Enable string oid type support for the GIE GAIA engine.
    • Support index query for the GAIA engine.

    2. More functions on persistent storage of GraphScope

    • Support loading dataset from the OSS.
    • Enable local GC for the Groot storage.

    3. Bug fixes and other enhancements

    • Upgrade the NetworkX version from 2.6 to 2.8.
    • A new algorithm named voterank has supported in the analytical engine.
    • Fix some problems in the property cache of GIE GAIA IR-Core.
    • Support to return an empty map if no property in valueMap operation.

    Docker Image

    docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.14.0
    

    Commits

    • [772e10749]: Bump karma from 3.0.0 to 6.3.16 in /python/jupyter (#1694) (DongZe Li)
    • [c2f1bd4b8]: set snapshot as extra params to get graph data of correct version in groot (#1666) (shirly121)
    • [b9093fc6c]: Enable local GC (#1653) (Siyuan Zhang)
    • [982d467af]: Increase the duration (#1695) (Siyuan Zhang)
    • [65523c6ef]: Optimize clustering by ignoring computation of vertices that degree less than 2 (#1597) (Weibin Zeng)
    • [5ad8d521c]: fuse expand/getV with the following filters to support more patterns in match (#1671) (shirly121)
    • [26887b514]: Disbale redefined warning of string_view (#1693) (DongZe Li)
    • [10b231f40]: Fix dependabot alert caused by crossbeam(#1687) (Neng Li)
    • [d5d714068]: Make CI filtering consistent between main and pull requests (#1686) (Siyuan Zhang)
    • [27e033846]: Enhance the script finding process when launching GIE from locally build without installation (#1690) (Tao He)
    • [275547ec9]: Support return a dict for graph schema (#1682) (DongZe Li)
    • [6e872a6e8]: Fixes the failure of nightly CI. (#1663) (Weibin Zeng)
    • [39c572bfb]: Merge gs-jython and gs-lib into graphscope package (#1680) (DongZe Li)
    • [3622ba8af]: Upgrade required vineyard version (#1655) (Tao He)
    • [c4a2f9ed6]: Upgrade Kafka version (#1660) (Siyuan Zhang)
    • [f06721e56]: Fix forward algorithm test import error which test dir not contain __init__ file (#1657) (Weibin Zeng)
    • [c5462af43]: Fixes the uninitialized pointer error for string fragment (#1659) (Tao He)
    • [b5bb4408d]: Return empty map if no properties in valueMap() (#1656) (shirly121)
    • [ce91b0ccf]: [GIE-IR] Upgrade Gaia-IR with the new Pegasus Client/Service APIs (#1648) (BingqingLyu)
    • [dc256c1bf]: [BugFix] Fix install zetcd error with GO111MODULE="auto" and remove the limitation of go version (#1651) (Weibin Zeng)
    • [59c85b0c4]: [GAIA/engine] Merge latest version pegasus engine (#1632) (Neng Li)
    • [593537580]: [GAIA-IR] Support index query in IR (#1618) (BingqingLyu)
    • [b5cee9c63]: Implement createGraph on coordinator, add API to initialize builder using schema (#1641) (Tao He)
    • [fcda4a687]: Add ipython to the distributed docker image. (#1643) (Tao He)
    • [0a50ed13f]: Enable string oid type support for FFI wrapper for GIE. (#1638) (Tao He)
    • [8fe16119f]: Add GC snapshot functionality for groot (#1635) (Siyuan Zhang)
    • [87b5613a2]: Support valueMap() to fetch all properties of vertices (#1623) (shirly121)
    • [b452508c4]: Fixes the bug in AddColumn to graphs with string oid type. (#1627) (Tao He)
    • [76d33884e]: [GAIA-IR] Map KeyId back into KeyName for labels and properties (#1615) (BingqingLyu)
    • [fa5718c96]: Listen 0.0.0.0 address in GAIA frontend (#1609) (DongZe Li)
    • [0d68477f1]: fix the join bug (#1608) (wzbxpy)
    • [4b33307d0]: [Bug Fix] Fix property cache in IR Core (#1606) (BingqingLyu)
    • [e83e8d9bc]: Implement voterank app (#1552) (liulx20)
    • [b5fff8686]: Fix networkx.cnli forward and reorg forward algorithm tests (#1584) (Weibin Zeng)
    • [33546c92e]: [IR Runtime] IR TagOpt for more compact structure in Runtime (#1594) (BingqingLyu)
    • [55d31d8a2]: Gie sdk basic auth (#1579) (tianliplus)
    • [b8ee7f063]: Clean up include headers of analytical engine (#1546) (Weibin Zeng)
    • [0aa8c3c6f]: refactor gie sdk (tianliplus)
    • [0115a729a]: Reorganize and cleanup proto definitions. (#1547) (Siyuan Zhang)
    • [8fbf1929b]: add ci tests for ir on vineyard (#1515) (shirly121)
    • [6060a6b45]: Restrict protobuf version (#1549) (Weibin Zeng)
    • [fb81a8291]: Make client and coordinator install with develop mode in MakeFile (#1511) (Weibin Zeng)
    • [95bd70166]: Simplify the python code of networkx and reorganize test cases (#1540) (Weibin Zeng)
    • [cbe749cd3]: Upgrade NetworkX version from 2.6 to 2.8 (#1532) (Weibin Zeng)
    • [c03b5395d]: Mark operation evaluated after the op is run (#1528) (Weibin Zeng)
    • [4c16d5364]: add oss support (#1481) (tianliplus)
    • [d784b2512]: Upgrade graph-learn version (#1516) (Weibin Zeng)
    • [9357e1317]: Revise DynamicToArrow with existed engine schema of DynamicFragment (#1501) (Weibin Zeng)
    Source code(tar.gz)
    Source code(zip)
  • v0.13.0(May 6, 2022)

    We are delighted to announce the release of GraphScope v0.13.0. This release is focused on providing a Jupyterlab extension for GraphScope to make your graph computation workflows better. In addition, we continuously work on improving the performance of GraphScope, and developer usability. We highlight the following improvements included in this release:

    1. Introduce a JupyterLab extension for GraphScope. Currently, it supports the following functions

    • Provide a graphical user interface for monitoring the status of graphscope resources (e.g., session and graph)
    • Support to define data schema and load graph in an interactive way
    • The extension has been integrated into our PlayGround

    2. Performance improvement

    • Add data caching in NetworkX for improving the performance of graph reporter
    • Optimize performance of arrow fragment to dynamic fragment with multiple thread

    3. BUG Fixes or other enhancements

    • Add a lock to protect the coordinator to avoid data race conditions
    • Fixes compilation failure in cdlp with ArrowFlattenFragment
    • Attempt backoff and retry during download dataset
    • Support primary key index in Groot
    • Support avg_clustering builtin application
    • Display progress bar during loading graph with k8s mode

    Docker Image

    docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.13.0
    

    Commits

    • [d0de89a55]: Disable brew update when installing dependencies as it is super slow. (#1506) (Tao He)
    • [ac1c1eaf2]: Upgrade graph learn version and force pyarrow<=6.0.0. (#1503) (Tao He)
    • [db579a7ad]: Move networkx graph schema from client to engine (#1485) (Weibin Zeng)
    • [617ae6312]: Display progress bar in k8s mode (#1495) (Siyuan Zhang)
    • [b035d0a7e]: Fix broken convert test of networkx (#1496) (Weibin Zeng)
    • [c833e372d]: Upgrade vineyard to 0.4.1 in Makefile (#1493) (DongZe Li)
    • [091b698de]: Update ScaleUp trigger in HorizontalRunnerAutoscaler (#1489) (Siyuan Zhang)
    • [ae9a34601]: Sync with latest changes about arrow fragment in vineyard, and include several refactors/bugfixes. (#1487) (Tao He)
    • [6df93c212]: Fix remove nodes logic in DynamicFragment (#1484) (Weibin Zeng)
    • [a75f33c8a]: Init the op of nx.Graph with incoming graphscope graph's op (#1479) (Weibin Zeng)
    • [2bc40ad5b]: Add another write path that uses the C++ IO adaptor (#1480) (Siyuan Zhang)
    • [23b612bce]: Remove the unused append-only arrow fragment. (#1477) (Tao He)
    • [4453f2e2a]: Suppress the grpc warnings in fork_posix.cc. (#1478) (Tao He)
    • [8d5a02939]: Optimize performance of arrow fragment to dynamic fragment with multi-threading (#1458) (Weibin Zeng)
    • [daa8649d4]: Support avg_clustering builtin app in client side (#1475) (DongZe Li)
    • [f27689c93]: Fix arrow version in Dockerfile (#1476) (Siyuan Zhang)
    • [05b0c3685]: Add del to nx.Graph to handle graph delete (#1464) (Weibin Zeng)
    • [2745d2525]: Move maxgraph*.jar into gs-jython package (#1472) (DongZe Li)
    • [4de8e6cc2]: groot support pk index (#1471) (tianliplus)
    • [ca9c7ad80]: Add a lock in the coordinator's RunStep as well. (#1465) (Tao He)
    • [0479c0a96]: Adapt CI on self-hosted runner to HorizontalRunnerAutoScaler (#1462) (Siyuan Zhang)
    • [90876cc06]: Attempt backoff and retry during download dataset (#1463) (DongZe Li)
    • [bbc7ebc8d]: Correctly handle the lifecycle of graph (#1460) (Siyuan Zhang)
    • [d52346279]: Add clean target to MakeFile to clean temporary files create from building (#1459) (Weibin Zeng)
    • [708ab4db8]: Delete intermediate files (#1456) (tianliplus)
    • [c4d2aefd9]: Bump hadoop-common from 2.10.1 to 3.2.3 in /interactive_engine (#1450) (dependabot[bot])
    • [28ca22997]: Correctly handle the empty dag (#1457) (Siyuan Zhang)
    • [632af342c]: Release the jupyter image in CI workflow (#1453) (DongZe Li)
    • [ea9193174]: Fix build wheel fail on mac (#1452) (Weibin Zeng)
    • [2cdeb8ec3]: Add data caching for networkx to improve the graph report performance (#1369) (Weibin Zeng)
    • [332ca48e4]: Add version check to coordinator & client, fixes #1435 (#1441) (wuyueandrew)
    • [9e2d77d83]: Fixes compilation failure in cdlp and sssp_path (#1447) (DongZe Li)
    • [46b30e450]: Add a lock to protect the coordinator to avoid data race condtions. (#1445) (Tao He)
    • [8af17dfb3]: Shrink gs-lib package to less than 100MB (#1440) (DongZe Li)
    Source code(tar.gz)
    Source code(zip)
  • v0.12.0(Apr 2, 2022)

    The GraphScope v0.12.0 release is a major update on many aspects of the project including backend engines, APIs, and system stability. It introduces an intermediate representation (IR) layer into the graph interactive engine (GIE) GAIA, to decouple query languages from query execution engines. Meanwhile, this release supports Giraph APIs to allow Giraph apps running on the Graph Analytics Engine (GAE) of GraphScope.

    We highlight the following improvements included in this release:

    1. Introduce IR layer into GAIA:

    • A completely redesigned IR layer to decouple the query language dependency, and further for query optimizations.
    • Define the supported Gremlin's grammar via Antlr.
    • Support match step in Gremlin, for graph pattern matching.

    2. Add Giraph APIs to GAE:

    • Support to load graphs with Giraph Formats, e.g., graph = sess.load_from(vertices="p2p-31.v", vformat="giraph:com.example.vformat",edges="p2p-31.e", eformat="giraph:com.example.vformat")
    • Support Giraph APIs on GraphScope, and all algorithms implemented towards the original Giraph APIs can run on GraphScope without modifications. For example, to load Giraph app SSSP, users can just invoke giraph_sssp = load_app(algo="giraph:com.alibaba.graphscope.example.giraph.SSSP")
    • Users now can add jars by sess.add_lib interface.

    3. Bug Fixes or other enhancements:

    • Correct the lifecycle management of loaded graphs within a session.
    • Make the graphlearn and tensorflow related logs more user-friendly.
    • Fix readwrite failed on GraphScope Networkx module.
    • Enable to connect to the existed ETCD cluster in GraphScope.
    • Support to launch GraphScope on K8s from an inner-pod environment.

    Docker Image

    docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.12.0
    

    Commits

    • [b0a090082]: Support launch GraphScope on K8s from inner-pod environment (#1434) (DongZe Li)
    • [9e4e47f15]: Add offline deployment doc for Helm (#1433) (DongZe Li)
    • [477931537]: Migrate cargo rocksdb dependency to 0.18.0 version (#1430) (Zichao Zhang)
    • [2e74e3c47]: Split the builtin application distribution to 'gs-apps' package (#1429) (DongZe Li)
    • [47b0d42d2]: Support string tensor for tensor context (#1425) (Weibin Zeng)
    • [1068628fe]: Upgrade the version of databind, fixes the dependent bot warnings. (#1424) (Tao He)
    • [ab774812e]: Upgrade the required black version. (#1422) (Tao He)
    • [92c4b155e]: Attempt backoff and retry during handle GRPC error (#1421) (DongZe Li)
    • [2de6b7e93]: Bump commons-io from 2.6 to 2.7 in /research/query_service/ir/compiler (#1417) (dependabot[bot])
    • [edb2493e4]: Retry 3 times and then report failure during HeartBeat process (#1419) (DongZe Li)
    • [93ef1b56c]: Try to fixes nightly CI on macOS by packaging 'string_view' into wheel package (#1420) (DongZe Li)
    • [9d2c1e775]: Bump jackson-databind from 2.11.1 to 2.13.2.1 in /interactive_engine (#1415) (dependabot[bot])
    • [64c77c13c]: Add networkx tutorial and fixed vineyard version (#1412) (DongZe Li)
    • [27400d1f1]: Bump protobuf-java in /research/query_service/ir/compiler (#1416) (dependabot[bot])
    • [7d2f92c93]: Introducing IR query service (#1407) (BingqingLyu)
    • [88e767069]: [BugFix] Fix analytical engine can't not compile with NETWORKX=OFF (#1410) (Weibin Zeng)
    • [ed6546da0]: Push if condition down to steps to allow it be checkable. (#1404) (Tao He)
    • [07cb0dff7]: Remove unused dependencies (#1403) (Zhang Lei)
    • [bf6998587]: Fix networkx forward nightly ci (#1387) (Weibin Zeng)
    • [0191dd62e]: Add dummy CI worflows to allow all status checkable. (#1402) (Tao He)
    • [f5928a344]: Giraph on GraphScope implementation (#1386) (Zhang Lei)
    • [e851014bc]: Merge lastest version of pegasus (#1376) (Neng Li)
    • [0bf6686d9]: Upgrade vineyard to v0.3.21. (#1389) (Tao He)
    • [3b520cab0]: Remove gaia from GraphScope for code refactor (#1388) (shirly121)
    • [91f4248c7]: Add role and rolebinding for graphscope charts (#1382) (DongZe Li)
    • [c921cd622]: Try to upgrade tinkerpop version to 3.5.1 in maxgraph (#1349) (shirly121)
    • [a314bbbe6]: Revert "Giraph on GraphScope (#1362)" (#1385) (Tao He)
    • [c3903d5ee]: Refactor DynamicFragment with CSREdgecutFragmentBase and MutableCSR (#1328) (Weibin Zeng)
    • [0a9cda988]: Bump protobuf-java from 3.18.0 to 3.18.2 in /interactive_engine (#1381) (dependabot[bot])
    • [83ebed579]: Upgrade proto to 3.18.0 && grpc to 1.42.1 (#1149) (shirly121)
    • [ee81be7d6]: Bump zookeeper from 3.4.5 to 3.4.14 in /analytical_engine/java (#1372) (dependabot[bot])
    • [1dbc3735e]: Giraph on GraphScope (#1362) (Zhang Lei)
    • [4d8390aba]: Disable noisy log inside gremlin-python, and fixes the hard-coded DEBUG. (#1368) (Tao He)
    • [2c56d49f3]: Enable to connect to the existed ETCD cluster in GraphScope (#1359) (wuyueandrew)
    • [5ea7c0de7]: [Bug fix] Fix readwrite test failed on graphscope.nx (#1365) (Weibin Zeng)
    • [c72da15fe]: Makes the graphlearn- and tf- related logs looks better. (#1367) (Tao He)
    • [9d95f32e9]: Frontend async rpc (#1357) (tianliplus)
    • [b61457c5c]: Upgrade version of formatter (#1363) (Siyuan Zhang)
    • [79904fc8a]: Correct the lifecycle of graph with session (#1361) (DongZe Li)
    • [9f29a7bb2]: Catch unknown exceptions in gRPC handlers. (#1358) (Tao He)
    • [cf4284e1f]: Also tag latest when release new version (#1360) (Siyuan Zhang)
    • [8a81ea26a]: The GAIA ci should be disable months ago. (#1356) (Tao He)
    • [e4040dba0]: It doesn't make sense to trigger those CI actions on forks. (#1354) (Tao He)
    • [4f2a2e688]: Add an env USE_GAIA_ENGINE to allow testing gaia from Python client. (#1348) (Tao He)
    Source code(tar.gz)
    Source code(zip)
  • v0.11.0(Mar 2, 2022)

    We are glad to announce a number of new features and improvements to GraphScope, alongside the GraphScope v0.11.0 release. This major release introduces mutable graphs into GraphScope, and adds GPU supports for the graph analytics engine (GAE). It also focuses on user-friendly improvements, code quality, and a series of bug fixes.

    We highlight the following improvements included in this release:

    Supporting mutable graphs:

    • Providing a set of interfaces (e.g., add_vertex, add_edge and update_vertex) to modify the topology of an existing graph.
    • Adopting existing apps to mutable graphs.

    Accelerating graph analytics apps with GPUs:

    • Adding new fragment, worker, message_manager and apps to utilize GPUs to process graph analytics tasks faster.
    • On top of these new infrastructures, 6 example algorithms (BFS, WCC, SSSP, PR, CDLP, LCC) are implemented.

    Optimizations and enhancements:

    • Stream grpc request(response) to support loading graph(fetching result) from(to) numpy/pandas more than 2GB.
    • Accelerating manipulation of graph topology by replacing folly::dynamic with rapidjson::value.

    https://graphscope.io/blog/releasenotes/2022/03/06/release-notes-0.11.0.html

    Docker Image

    docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.11.0
    

    Commits

    • [f30b95c7]: Fixes CI failure on macOS (#1346) (DongZe Li)
    • [d911efe5]: Stop coordinator gracefully by accepting the signal from notebook restarted/stopped event (#1342) (DongZe Li)
    • [ff33f5a6]: Keep the global symbol to avoid it been stripped on Mac. (#1345) (Tao He)
    • [51f680aa]: Upgrade required vineyard version to 0.3.19 (#1339) (Siyuan Zhang)
    • [569c04b2]: Fixes PATH to not include $LLVM/bin to make sure the correct ranlib be found. (#1341) (Tao He)
    • [0b083e54]: Revert "Make sure the $PATH could search /usr/bin first. (#1340) (Tao He)
    • [158f4525]: Make sure the $PATH could search /usr/bin first. (#1338) (Tao He)
    • [385accd1]: Extract a util dynamic library from grape-engine (#1337) (Siyuan Zhang)
    • [e3f055ec]: Local bench groot (#1326) (tianliplus)
    • [953cc3e0]: Add timeout after launch vineyard to ensure we catches the error code correctly (#1332) (Siyuan Zhang)
    • [5e33f669]: Improve the docs. (#1324) (Tao He)
    • [e13eea9c]: Adapt to mutable fragment. (#1320) (luoxiaojian)
    • [f14b6f65]: Bump up graph-learn package for checkpoint. (#1323) (Tao He)
    • [ec3f3ef7]: Use LargeAttrValue in modify operation and dumps json with orjson (#1321) (Weibin Zeng)
    • [1a94a810]: Enable download metadata for repo 'appstream' of centos-8 (#1318) (DongZe Li)
    • [ec0ac9de]: Stream RunStepResponse of grpc to support fetching result more than 2GB (#1314) (DongZe Li)
    • [156c1c83]: Replace folly::dynamic with rapidjson::Value and bump vineyard to 0.3.18 (#1271) (Weibin Zeng)
    • [f3113bac]: Bump hadoop-common from 2.8.4 to 2.10.1 in /interactive_engine (#1312) (dependabot[bot])
    • [6918b764]: Stream RunStepRequest of grpc to support loading graph from numpy/pandas more than 2GB (#1309) (DongZe Li)
    • [cdb0dcfa]: Use boost::lexical_cast to handle string id fragment (#1310) (Siyuan Zhang)
    • [3b607ec2]: Bump httpclient in /research/query_service/gremlin/compiler (#1306) (dependabot[bot])
    • [0200ceb8]: Bump commons-io in /research/query_service/gremlin/compiler (#1307) (dependabot[bot])
    Source code(tar.gz)
    Source code(zip)
    data_load-f486c559423c757ca775074ba2ba26b4c25af323.zip(106.03 MB)
  • v0.10.1(Jan 29, 2022)

    We highlight the following fixes and improvements included in this releases:

    • Support running built-in app on the property graph without project.
    • Support running UDF app on the property graph with string vertex id.
    • Improve the document of the learning graph.
    • Drop support for Python 3.6

    Docker Image

    docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.10.1
    

    Commits

    • [97d12bf5]: Fixes failure on macOS CI (#1304) (DongZe Li)
    • [10f25808]: Add 'large_attr' data structure in op definition for large chunk (data of pandas/numpy) (#1303) (DongZe Li)
    • [e554e936]: [networkx] binding session from graphscope graph and fix bug (#1302) (Weibin Zeng)
    • [3dc75f49]: Better schema representation (#1299) (Siyuan Zhang)
    • [66b74fd4]: Cython sdk rst: Context.superstep() change return type str -> int (#1296) (wuyueandrew)
    • [7eeb7259]: Print data type in graph schema (#1294) (Siyuan Zhang)
    • [66825203]: Check the nullptr of graph_info (#1295) (Siyuan Zhang)
    • [7fc69d99]: Fix: nightly CI was broken on test_error_on_selector case (#1291) (DongZe Li)
    • [fd676d77]: Fix the invalid identifier 'FULL-TEST-SUITE' in the environment variable (#1289) (DongZe Li)
    • [26f52147]: Support running udf app on property graph with string oid type (#1287) (DongZe Li)
    • [b683dcd7]: Raise an error and tell users what to do on Windows. (#1285) (Tao He)
    • [e8f7aaf4]: Improve the document of learning.Graph (#1277) (Weibin Zeng)
    • [976345bf]: Fixes building docs for graphlearning, and the scroll bar. (#1279) (Tao He)
    • [06e3392e]: Support running built-in app on the property graph by projecting to flattened fragment implicitly (#1258) (DongZe Li)
    • [0ba3d5e8]: Drop support for Python 3.6 (#1268) (Weibin Zeng)
    • [d3e85456]: Fix the broken networkx ci (#1264) (Weibin Zeng)
    • [1446cd6d]: Parallelize eigenvector_centrality and katz_centrality (#1262) (Weibin Zeng)
    • [38470b0d]: Revise built-in clustering and replace forward with built-in (#1255) (Weibin Zeng)
    • [334e6248]: Bump protobuf-java from 3.5.1 to 3.16.1 in interactive_engine (#1260) (dependabot[bot])
    • [83c68425]: Guard folly headers when NETWORKX=OFF (#1259) (Siyuan Zhang)
    • [a45a7b99]: Optimize modification APIs of graphscope.nx (#1250) (Weibin Zeng)
    • [1ae368f3]: Add metrics for benchmark groot (#1238) (tianliplus)
    Source code(tar.gz)
    Source code(zip)
  • v0.10.0(Jan 5, 2022)

    We are glad to announce the availability of GraphScope v0.10. This release supports users to run GraphScope on Macs powered by Apple's new M1 chip. In addition, it allows to serialize/ deserialize graph data to/from the disk under the standalone mode.

    We highlight the following improvements included in this release:

    • Add Apple's M1 chip support.
    • Improve the implementation and documentation of context selectors.
    • Support serialization/deserialization of graph data on the standalone mode.
    • Refactor Java SDK in Graph Analytics Engine.
    • Fix a bug when loading graph data from HDFS.
    • Fix got empty result in all_simple_path algorithm with num_workers=1

    Docker Image

    docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.10.0
    

    Commits

    • [c49b1b21]: Format java source code of analytical engine (#1252) (DongZe Li)
    • [306da050]: Refactor GAE Java (#1227) (Zhang Lei)
    • [64afab7d]: Fixed the auditwheel version to 5.0.0 (#1251) (DongZe Li)
    • [847a6a29]: Quickfix: a syntax error. (#1248) (Tao He)
    • [38ad3def]: Fixes the long-standing "Event loop is closed" issue. (#1247) (Tao He)
    • [53316583]: Bring back mars support, re-enable mars test in CI, and fixes several related issues. (#1246) (Tao He)
    • [9b32ccfc]: Enable serialization test on K8s, and add a new case to test local serialization (#1245) (Tao He)
    • [819b9709]: Enable customization of storage class (#1244) (Siyuan Zhang)
    • [038e915a]: Print diffs when failed to format java source code. (#1243) (Tao He)
    • [529ecd22]: Consistent label checking in client side (#1242) (Siyuan Zhang)
    • [a7d1472f]: Remove unrelated file from gh-pages (#1241) (Siyuan Zhang)
    • [89f03da1]: Add documents about potential permission error when allocating PV in NFS. (#1239) (wuyueandrew)
    • [db9b427d]: Format java code with google style (#1233) (DongZe Li)
    • [e37f69c1]: Upgrade log4j2 (#1235) (tianliplus)
    • [e1e22a42]: Fix log not fetched to client (#1236) (Siyuan Zhang)
    • [2e7e6399]: Changes on property applications (#1228) (DongZe Li)
    • [ab75ad71]: Fix error when selecting from empty graph (#1231) (Siyuan Zhang)
    • [9f8e7f21]: etcd: Increate the ETCD pods period seconds (#1213) (Hao Xin)
    • [0c60e6de]: Bump up required vineyard version to v0.3.14 and reable HDFS IO tests. (#1229) (Tao He)
    • [84ca6cd8]: Fix some typo in persistent_graph_store.rst (#1232) (wuyueandrew)
    • [e26e5330]: Fixed a typo in persistent_graph_store.rst (#1226) (wuyueandrew)
    • [06de1f17]: Make forward nightly ci pass (#1223) (Weibin Zeng)
    • [5b724b7a]: Fix codecov coverage ratio (#1224) (DongZe Li)
    • [a8129084]: Make GraphScope support on Apple M1 chip (#1221) (DongZe Li)
    • [f2ed9a14]: Typo of conn (#1222) (Siyuan Zhang)
    • [9e912363]: Upload codecover report only in ubuntu20 platform (#1220) (DongZe Li)
    • [d7140119]: add edit_on_github button on docs. (#1214) (Jingbo Xu)
    • [1eacd877]: Make the definition of src in sssp and bfs accurate (#1218) (Siyuan Zhang)
    • [ed4a82b3]: Report codecover of nightly CI (#1219) (DongZe Li)
    • [62c0676c]: [networkx] Revise the add_node/edge with cache and upgrade NetworkX dependency to 2.6 (#1208) (Weibin Zeng)
    • [a6099b69]: Enhanced input validation and schema representation (#1205) (Siyuan Zhang)
    • [8cdc1b6b]: Rename occurances of "libvineyard" to "v6d". (#1202) (Tao He)
    • [ec000ba9]: Support compiling analytical engine on Apple M1 chip (#1200) (DongZe Li)
    • [76d4641e]: [networkx] Update the built-in app implementation to make the behavior consistent with NetworkX implementation (#1176) (Weibin Zeng)
    • [3049ef92]: Update codecov/codecov-action to v2. (#1193) (Tao He)
    • [1c62d2a4]: Increase the default resource in helm deployment (#1189) (DongZe Li)
    Source code(tar.gz)
    Source code(zip)
  • v0.9.1(Dec 19, 2021)

  • v0.9.0(Dec 3, 2021)

    We are glad to announce the availability of GraphScope v0.9.

    In this release, we revisit the Dev-infra to improve the productivity. Now, you can enjoy GraphScope with standalone mode both in our PlayGround and Google Colab.

    We also aims to continuously make GraphScope more user-friendly, and revised the documents and tutorials based on the latest version.

    Besides, we alsh have a preliminary support for Java in Graph Analytics Engine (GAE), and users can succinctly develop graph analytics applications with Java.

    For Graph Interactive Engine (GIE), we have done a lot of work to improve the performance of the distributed GAIA engine, which includes but is not limited to 1) simplifying the communication protocol that reduces many useless tags in correlated subtask; 2) making early-stop mechanism more effective due to 1); 3) resolving a lot of bugs due to 1); 4) refining the engine apis. As a distributed cyclic data-parallel engine, GAIA engine is in de-facto generic to handle any data-intensive task. Now users can assess to more GAIA APIs (will be well documented) to play with the engine.

    Features

    • Support GLE on MacOS
    • Java SDK in GAE, and examples to write and run algorithms written in Java.
    • Add several popular datasets and ships datasets by adding an extra container in Kubernetes mode.
    • Supplement NetworkX generator, read, write, drawing, convert with more generators and test cases.
    • Add make install support for lgraph.
    • GIE Runtime refactoring based on the new GAIA engine.
    • Reimplement persistent storage engine with zero-copy read.
    • Add common labels following Kubernetes best practice to GraphScope cluster.
    • Add transformation ability for directed/undirected graphs.
    • Unify the logging output file location to /var/log/graphscope or $HOME/.local/log/graphscope

    Bug Fixes

    • Fixed a bug that caused a crash if duplicate property names were encountered when extracting subgraphs.
    • Fixed MPI library not found problem on MacOS.
    • Fixed a data-lost bug in realtime write caused by kafka retention policy.
    • Fixed context processing on graphs generated by add_column.
    • Fixed a failure when exposing gremlin service deployed with Helm Charts.
    • Fixed many bugs related to subtasks while querying Gremlin.
    Source code(tar.gz)
    Source code(zip)
    graphscope_store_data_load.zip(98.79 MB)
  • v0.8.0(Nov 8, 2021)

    We are glad to announce the GraphScope 0.8 release. The GraphScope 0.8 release is a major update on many aspects of the project including deployment, system speed, and APIs. For quickly getting started, this release supports enjoying GraphScope on standalone mode without Kubernetes. To improve the efficiency of operators and applications in the NetworkX module, an immutable graph is applied by default, while it is converted to a dynamic graph only if modification operators for graphs are triggered. In addition, a jupyter notebook is integrated into the helm charts.

    We highlight the following improvements included in this release:

    1. Standalone mode support for GraphScope
    • User now can install graphscope by pip install graphscope or install from wheel manually to enjoy GraphScope on local with runtime dependencies
    • User now switch to pip install graphscope_client to install python client of GraphScope
    • MacOS support is added, and compatible with Apple clang 13.0.0
    1. Enhancement on graphscope.nx module
    • Support NetworkX operators over immutable graphs:
    • Support holding immutable graph in nx.Graph and copy-on-write to dynamic graph
    1. Enhancement of deployment, performance and APIs:
    • Integrate a jupyterlab notebook into the helm charts

    For more detailed improvements that have been made in this release, please refer to the complete changelog.

    Commits

    • [[7cfe16b7]: Don't export /usr/local/opt/llvm/lib to linker search path on MacOS. (#936) (Tao He)
    • [[52ebbaae]: Fix slf4j StaticLoggerBinder loading error (#941) (Zichao Zhang)
    • [[e52b1e1d]: Fixes helm charts jupyterlab container: use release version as image tag (#935) (DongZe Li)
    • [[6e5404ed]: Add readiness probe to graphscope-store frontend. (#937) (Siyuan Zhang)
    • [[a68c83ca]: [networkx] Riase NetworkXError when gs graph is multigraph (#924) (Weibin Zeng)
    • [[c80a2df5]: Add maxgraph backup module (#921) (Zichao Zhang)
    • [[476f3aab]: Cancel previous wheel packaging actions in CI as well. (Tao He)
    • [[a5eb44b3]: Support using GraphScope on standalone mode (MacOS) (#923) (DongZe Li)
    • [[5d4c84d6]: Cancel workflows defined in other yamls. (Tao He)
    • [[688f73a3]: Install LLVM and fastFFI in runtime/gsvineyard docker image. (#932) (Tao He)
    • [[c7815b03]: Cancel previous runs in PRs. (Tao He)
    • [[30658fd1]: Integrate a Jupyter-lab container in the graphscope helm chart. (#928) (Siyuan Zhang)
    • [[d5821a12]: Add a flattened wrapper on property graph to run apps defined on simple graph (#888) (Weibin Zeng)
    • [[a29a964f]: Unload unreachable op after run (#916) (Siyuan Zhang)
    • [[0bc8393c]: Remove unused parameters (#912) (Siyuan Zhang)
    • [[959cb731]: Schedule package CI as nightly run (#908) (DongZe Li)
    • [[cf00cf34]: Add unittest for global store ffi and fix an issue in global store ffi that uses a shared global client to access vineyard. (#905) (zhiminch)
    • [[e6bdb109]: Fixes permission denied in nightly CI (#906) (DongZe Li)
    • [[9291e5c3]: Fix groot ut (#903) (tianliplus)
    • [[d970e885]: Add backup module of graph store (#901) (Zichao Zhang)
    • [[8004e224]: [networkx] Implement average degree connectivity app (#516) (Xinger)
    • [[9ac63094]: Fixes nightly CI failure and symlink problem (#899) (DongZe Li)
    • [[3b1de8f3]: Add progress bar for loading graphs. (#894) (Siyuan Zhang)
    • [[dbe5d0b8]: reorg gie submodules (#893) (tianliplus)
    • [[4cdc4848]: Change rust-rocksdb dependency for store (#892) (GoldenLeaves)
    • [[e5fbefe2]: Separate download tests (#895) (Siyuan Zhang)
    • [[1c60ff1a]: Support using GraphScope on standalone mode (Linux) (#860) (DongZe Li)
    • [[308f35e0]: Reorg groot (#887) (tianliplus)
    • [[d91f55e7]: [networkx] Support reporter API on arrow fragment (#866) (Weibin Zeng)
    • [[2956abc1]: [networkx] Implement all simple paths app (#574) (MJY-HUST)
    • [[dd816c39]: Create and use default session when there is no default session found (#880) (DongZe Li)
    • [[26d320b0]: Fix unit tests under frontend module (#883) (shirly121)
    • [[5f25d518]: update kind download url (#882) (Weibin Zeng)
    • [[db3b5e2a]: Launch etcd in coordinator (#878) (DongZe Li)
    • [[1dac7f41]: [networkx] implement (weighted) betweenness centrality in graphscope. (#666) (mengke)
    • [[19245211]: [Bug Fix] fix bug in starting GAIA service in v2 and vineyard (#870) (BingqingLyu)
    • [[f47e0197]: Fix compilation warning. Remove utilities in vineyard store ffi. (#861) (zhiminch)
    • [[79818472]: [bug fix] support id-only vertex in EdgeV Step (#871) (BingqingLyu)
    • [[2cd20e58]: Require vineyard (for serialization test, etc.), but only on x86_64. (#873) (Tao He)
    • [[46e34e46]: Launch vineyard in a nicer way when vineyardd not found. (#872) (Tao He)
    • [[50946714]: address maven warnings by specifying the dependency versions. (#864) (Jingbo Xu)
    • [[51379546]: merge gie and groot schema api (#825) (tianliplus)
    • [[16d41fb5]: Revisit the bindings to mars. (#863) (Tao He)
    • [[d6616cd2]: Make the add_edges behavior more Pythonic (#855) (Siyuan Zhang)
    • [[bc5c430f]: Fix unit test in compiler of interactive engine (#859) (shirly121)
    • [[c972b605]: [networkx] Implement attribute assortativity app and numeric assortativity app (#515) (Xinger)
    • [[ce439b5c]: [networkx] Support label on DynamicFragment (#822) (Weibin Zeng)
    • [[13b6b4f8]: Support download dataset from origin URL in standalone mode (#828) (DongZe Li)
    • [[4bd128b4]: Fix an incorrect error initialization (#827) (Siyuan Zhang)
    • [[bf7204f9]: Fix: add data cache clean in gremlin_test.sh (#823) (Weibin Zeng)
    • [[9d795761]: Update local deploy script: upgrade jdk tool-chain and open macOS CI (#758) (Weibin Zeng)
    • [[1c99a983]: [networkx] Check if node is illegal and raise NetworkXError (#817) (Weibin Zeng)
    • [[febd633d]: Nightly: don't run on forks. (#818) (Tao He)
    • [[d21de371]: Recover log4rs.yml of graphscope-store and fix the log output location (#814) (Weibin Zeng)
    • [[1858a873]: GaiaX Runtime refactoring (#812) (BingqingLyu)
    • [[ee03c8fc]: Unified directory structure after installation (#809) (DongZe Li)
    • [[846dd4f5]: Enable math.h functions in UDF algorithms. (#811) (Tao He)
    • [[60ab3af1]: Gaia x (#801) (Longbin Lai)
    • [[7dc8086d]: Add parameters section to Store docs (#802) (Siyuan Zhang)
    • [[bc048d79]: fix bug in graph store doc (#799) (tianliplus)
    • [[3e2fd7dd]: Add methods to unload contexts. (#794) (Siyuan Zhang)
    • [[b5ba16ca]: clear duplicated and unused code (#777) (tianliplus)
    • [[4b6b665a]: shutil.rmtree cannot remove files (#793) (Siyuan Zhang)
    • [[fce08189]: rename 0.0.1-SNAPSHOT as maxgraph-assembly-0.0.1-SNAPSHOT and update reference (#783) (shirly121)
    • [[c6fd2780]: Enable manually trigger nightly ci (#789) (Weibin Zeng)
    • [[106078ff]: Increase the resource size of coordinator for running demo of README (#791) (DongZe Li)
    • [[b03e12ba]: Build app/graph frames: with "Release" profile by default. (#790) (Tao He)
    Source code(tar.gz)
    Source code(zip)
    graphscope-0.8.0-py2.py3-none-macosx_10_9_x86_64.whl(203.92 MB)
    graphscope-0.8.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl(380.38 MB)
    graphscope_store_data_load.tar.gz(96.22 MB)
  • v0.7.0(Sep 8, 2021)

    We are glad to announce the availability of GraphScope v0.7. This release includes major updates for the persistent graph store in GraphScope, providing APIs for real-time graph updates (inserts and deletes of individual vertices and edges). It also focuses on user-friendly improvements, security issues, code quality, and a series of bug fixes.

    We highlight the following improvements included in this release:

    1. Apart from bulk loading, this release introduces a set of APIs for real-time graph updates. Currently, these APIs have supported the following functions:

      • Insert/delete one or multiple vertices/edges;
      • Update properties of a specific vertex/edge.

      More details can refer to this.

    2. User-friendly improvement

      • Revise error handling in GraphScope and improve all error messages reported to users;
      • Add a document to describe persistent graph store in GraphScope;
      • The logs in the err channel are always fetched to the client for debugging;
      • The bulk-loading tool of the persistent graph store is released to help load graphs into the store;
      • Revise some descriptions for APIs in documents.
    3. Optimizations and enhancements

      • Using zetcd to replace zookeeper in the graph interactive engine;
      • Update third-party dependencies to address some security issues;
      • More test coverages for GAIA and client.
      • Integrate GIE GraphManager into Coordinator.
      • During sess.gremlin, the pod will not be created dynamically for reducing the response time

    Some Breaking API Changes:

    • Remove GIE GraphManager role.
    • Remove zookeeper and replace with zetcd.
    • k8s_gie_graph_manager_image k8s_gie_graph_manager_cpu k8s_gie_graph_manager_mem Deprecated.
    • k8s_zookeeper_image k8s_zookeeper_cpu k8s_zookeeper_mem Deprecated.
    • k8s_gie_gremlin_server_cpu k8s_gie_gremlin_server_mem Deprecated.

    For more detailed improvements that have been made in this release, please refer to the complete changelog.

    Commits

    • [82c9aab3]: Fixes io loop event exeception by closing gremlin python client (#788) (DongZe Li)
    • [b152c3e2]: Fix inaccurate internal error code (#787) (Siyuan Zhang)
    • [dc8f214c]: Fix restart test of GraphScope-Store(#784) (tianliplus)
    • [42a9e8ef]: Fixes the condition of whether vineyard exists. (#782) (Tao He)
    • [352be042]: Install GraphScope into /opt/graphscope and link to the /usr/local (#767) (DongZe Li)
    • [74b13bf1]: Vineyard is no longer a hard dependency for the client package. (#780) (Tao He)
    • [684522a4]: Revise error messages and handling of GRPC exception. (#751) (Siyuan Zhang)
    • [f4905b2f]: add doc for persistent graph store (#765) (tianliplus)
    • [54e32adc]: Config coordinator logger to handle warning level to stderr (#764) (Weibin Zeng)
    • [a0a1165e]: Fixes the sphinx and docutil version to make sure lists are rendered. (#766) (Tao He)
    • [8e21257c]: Fix pegasus service to run multiple executors (#754) (shirly121)
    • [845ba6d0]: Output the graphscope-store log to both console and file (#749) (Weibin Zeng)
    • [96fba3d1]: Add a ci to test loading graph from HDFS (#750) (Weibin Zeng)
    • [1e602fc2]: Remove the legacy code related to graph manager and make default user 'graphscope' in graphscope-store image (#748) (DongZe Li)
    • [e6861434]: Fetch stdout and stderr log of grape engine separately (#744) (Weibin Zeng)
    • [22db6d95]: Enables GAIA to run on the Kubernetes environment (#732) (DongZe Li)
    • [4508158c]: Fix edge id encoding for v2 (#743) (shirly121)
    • [9e8e066d]: Implement realtime write API in Python side, and add tests in CI. (#721) (Siyuan Zhang)
    • [e013e3b1]: Add unified Tinkerpop version variable to reduce duplicate constant in pom (#735) (shirly121)
    • [741ccfe4]: Provide a method to print human readable schema for Context. (#734) (Siyuan Zhang)
    • [434b8729]: Upgrade tinkerpop & groovy (#714) (shirly121)
    • [f1307ef7]: Replace zookeeper with zetcd+etcd (#722) (Weibin Zeng)
    • [61bdc84a]: Allow the session to be reconnectable, even not reaching dangling timeout (#733) (Tao He)
    • [0b341e17]: Update README by adding publications. (Jingbo Xu)
    • [fd2d0429]: Push docs for tags as well. (#728) (Tao He)
    • [4c82d024]: Remove unused files (#720) (BingqingLyu)
    • [9e9a52de]: Add notes for self-hosted runner (#723) (Siyuan Zhang)
    • [cea8fb1a]: Fix gremlin test hang by move gss-test to self-hosted runner (#717) (tianliplus)
    • [ea450f03]: Fix store warning (#718) (tianliplus)
    • [d18a82e8]: Improve Cython API doc by adding 'graphscope.declare' 'graphscope.Vertex' in cython_sdk.rst (#719) (DongZe Li)
    • [07ed08ea]: Make I/O a lazy DAGNode in the coordinator (#715) (DongZe Li)
    • [358fc7de]: Fixes for CXX_FLAGS detection on Linux with LLVM compiler. (#716) (Tao He)
    • [fe4837b7]: Adapt to latest vineyard: the Create() API has changed. (#713) (Tao He)
    • [9405412f]: Add GAIA on vineyard test (#708) (shirly121)
    • [e0927741]: Implement degree assortativity coefficient app (#492) (Xinger)
    • [3105127a]: Missing close method in RPC client. (#704) (DongZe Li)
    • [fe37bee5]: remove unused file (#705) (shirly121)
    • [c07654de]: Remove dependency of v2 from gaia-adaptor (#702) (shirly121)
    • [66706e72]: Fix docker-nightly ci with short sha (#703) (Weibin Zeng)
    • [b1c914a0]: Fix variable names when starting GIE instance (#700) (Siyuan Zhang)
    • [14eee011]: Update cmake minimum version to 3.1 for gie (#699) (Weibin Zeng)
    • [ce261cb8]: An ad-hoc solution to find frontend port from log file (#693) (Weibin Zeng)
    • [6ce1e8ac]: Add BiparteGraphSage tutorial to show use of categorical_attrs_desc (#691) (Weibin Zeng)
    • [f8fa4f6f]: Fix docker nightly ci: schedule time and gstest clean (#688) (Weibin Zeng)
    • [16c0c722]: Upgrade the graphlearn submodule to fixes a "get attribute" bug in GLE. (#687) (Tao He)
    • [521e1957]: Fix failure of launching learning engine by allowing pickle None value (#686) (DongZe Li)
    • [c4355911]: remove pom (#680) (tianliplus)
    • [35b2fdfe]: Improve calculateQ efficiency in Louvain (#614) (Siyuan Zhang)
    • [79ec8008]: Upgrade dependency (#674) (shirly121)
    • [40ad8cfd]: fix sending unvalidated data to a web browser (#671) (shirly121)
    • [3ec9bdb5]: Add a schedule ci to test docker file in k8s (#634) (Weibin Zeng)
    • [b9183682]: upgrade smallvec to 0.6.14 as recommended (#675) (BingqingLyu)
    • [8c28beb3]: support java16 && upgrade rust toolchain (#668) (tianliplus)
    • [c90a258a]: Fix warnings in pegasus and runtime in interactive engine (#670) (BingqingLyu)
    • [adb04c92]: refine gremlin test (#663) (tianliplus)
    • [bd877ffe]: upgrade crossbeam-channel from 0.3.x to 0.4.4 as suggested (#656) (BingqingLyu)
    • [fcca5f2a]: update image links (#658) (Siyuan Zhang)
    Source code(tar.gz)
    Source code(zip)
    graphscope_store_data_load.tar.gz(163.87 MB)
  • v0.6.0(Aug 6, 2021)

    We are glad to announce the release of GraphScope 0.6.0

    This major release integrates a new graph interactive engine GAIA, which supports efficient parallel execution and bounded-memory execution for gremlin queries. More technical details of GAIA can refer to our published blog. Note that currently the integration of GAIA with GraphScope is experimental, and is not recommended for production use yet!

    In addition, this release improves the experience of local deployment on macOS, Ubuntu, and CentOS, and adds more graph analytics algorithms.

    We highlight the following improvements included in this release:

    1. Integrate GAIA, a graph interactive query engine, into GraphScope. Currently, it has supported the following features/functions:

      • Dynamic memory management for arbitrary graph traversal with ensuring bounded use of memory.
      • Automatic and adaptive strategy for optimizing Gremlin traversals, such as hybrid DFS/BFS traversal to balance parallelism and memory usage.
      • Early-stop optimization for Gremlin (limit, nested conditional, etc.) to minimize wasted computation.
      • Improvement of performance and scalability. (A new LDBC Social Network Benchmark will be released around year-end).
      • Support both Vineyard and the new persistent graph store.
    2. Lazy evaluation support for graph interactive engine and graph learning engine.

    3. A script supporting local deployment on macOS, Ubuntu, and CentOS.

    4. Add more graph analytics algorithms as built-in applications.

    Commits

    • [032ed994]: Reorganize the scripts for GIE launching (#581) (Weibin Zeng)
    • [eb71eefe]: Add glossory to analytics_engine.rst (#657) (Siyuan Zhang)
    • [1d0632ad]: Add unit tests for gnn store ffi. (#654) (GoldenLeaves)
    • [1af190b6]: Add gremlin test (#644) (tianliplus)
    • [a383bbcd]: Upgrade to vineyard v0.2.6. (#655) (Tao He)
    • [e2987fa6]: Open CI of Vineyard X [Maxgraph + GAIA] (#633) (Siyuan Zhang)
    • [36d7bc49]: [Bug Fix] Fix bugs in CI of GAIA on Vineyard (#652) (BingqingLyu)
    • [4fe934a5]: Fix broken hadoop link (#651) (tianliplus)
    • [59703c8c]: fix (#648) (tianliplus)
    • [b2363f2c]: Fix bugs of negative vertex id (#640) (BingqingLyu)
    • [007a7a15]: Store ffi for GNN (#638) (tianliplus)
    • [2393760c]: Open GraphScope Store GAIA test (#597) (Siyuan Zhang)
    • [1998c818]: Fix data load (#627) (tianliplus)
    • [b114ce23]: [Bug Fix]: add u128 in Primitive in Object (#628) (BingqingLyu)
    • [b264c4c0]: [Bug Fix]: fix bug in path().count() (#629) (BingqingLyu)
    • [c4a77202]: bug fix (#615) (tianliplus)
    • [d5460387]: [script] Add scripts for starting GAIA on vineyard (#604) (BingqingLyu)
    • [d902f121]: remove pb gen (#601) (tianliplus)
    • [1f62da38]: [script] Start GAIA executor on Vineyard (#600) (BingqingLyu)
    • [b3b3a873]: [networkx] Implement is_simple_path app in GraphScope (#471) (MJY-HUST)
    • [576d8b80]: [Bug Fix] GAIA executor with vineyard (#598) (BingqingLyu)
    • [e47915fe]: add additional pegasus config for benchmark (#587) (shirly121)
    • [90ac841d]: [GAIA & Store Adapter] Support parallel scan in GAIA, and adapt to v2 and vineyard respectively (#592) (BingqingLyu)
    • [fefffbd7]: [GAIA] Start GAIA executor with vineyard store (#595) (BingqingLyu)
    • [99aca1ca]: Update deploy_local.sh: add CentOS support and more robust (#562) (Weibin Zeng)
    • [19f7d0b5]: add vineyard adaptor (#590) (shirly121)
    • [d1e935e2]: Add config matrix for testing with/without GAIA in CI (#572) (Siyuan Zhang)
    • [8b060084]: divide store (#579) (tianliplus)
    • [3de9a7a8]: Add gaia compiler adaptor into V2 to fix dependency issue (#576) (shirly121)
    • [ec4caff5]: Fix session.close for local test (#573) (DongZe Li)
    • [e84765f3]: Fix local CI by catching exception in sess.close (#571) (DongZe Li)
    • [0a7cd8f9]: [store_adapter] Adapt Partitioner for vineyard on Read Path (#560) (BingqingLyu)
    • [6ff99b52]: Lazy mode support for GIE/GLE engine (#513) (DongZe Li)
    • [076268d5]: Create CODE_OF_CONDUCT.md (#567) (Jingbo Xu)
    • [66bdc559]: Realtime write server impl (#557) (tianliplus)
    • [a8c5d129]: Use the same etcd prefix when launching vineyard from multiple engines. (#559) (Tao He)
    • [685965e6]: Install the inplace vineyard-io as well. (#558) (Tao He)
    • [9fdb14fd]: Simplify local ci with matrix.os (#556) (Weibin Zeng)
    • [ff23d9a1]: Add CI test for GAIA compiler (#552) (shirly121)
    • [76cbcee3]: Revise deploy_local.sh and update local ci with deploy_local.sh (#555) (Weibin Zeng)
    • [626a1059]: Fix edge result (#549) (shirly121)
    • [26436e96]: [WIP] Realtime write api (#553) (tianliplus)
    • [d2ab2597]: Fix macOS ci: change to llvm 8 (#554) (Weibin Zeng)
    • [1f7588a6]: Fix gss test bugs (#548) (shirly121)
    • [85432f65]: Unify rpc api (#547) (shirly121)
    • [338da433]: add realtime_write proto (#546) (tianliplus)
    • [ca65eda0]: implement GraphStoreService api for maxgraph && add snapshot (#537) (shirly121)
    • [e28c4416]: checkin store v2 interface (#543) (tianliplus)
    • [e280e59c]: Revise the prepare_env.sh, more standard and user-friendly (#534) (Weibin Zeng)
    • [1f30e04a]: Add a local deploy and test ci on MacOS (#512) (Weibin Zeng)
    • [af99e5bd]: use gremlin api in 3.4.2 to be compatible with maxgraph (#540) (shirly121)
    • [101ab6f3]: Unified Query Params for compatible with different storages (#527) (BingqingLyu)
    • [68c4cfb7]: Initialize job compiler in adapter for starting engine service (#530) (BingqingLyu)
    • [3571ca4a]: Separate the pytest in ci to avoid OOM (#529) (Weibin Zeng)
    • [55adea97]: [GAIA] Add storage adapter for graphscope store (#521) (BingqingLyu)
    • [ae6b8c5f]: [NetworkX] add node_boundary and edge boundary app (#468) (Weibin Zeng)
    • [33a4328f]: [store_adapter] refine some codes in store adapter (#519) (BingqingLyu)
    • [6fba7ca0]: Doc: sections in alphabetic order. (#520) (Tao He)
    • [7660da51]: Add extra params for compatible with different storages (#508) (BingqingLyu)
    • [73e05ba0]: fix reverse edge in realtime write (#510) (tianliplus)
    • [7e21a25e]: Fixes the copy_class hack for argmap in networkx. (#511) (Tao He)
    • [f1da6ae6]: Use a higher version git (#506) (Siyuan Zhang)
    • [d297b6bd]: Set decorator>=5.0.5 since it's old version(4.4.2) may not compatible with NetworkX latest release (#507) (Weibin Zeng)
    • [8cc4f26e]: Add examples for output context (#505) (Siyuan Zhang)
    • [bb9d1928]: Typo: we actually use BUILD_PROGRESS rather than DOCKER_BUILD_PROGRESS. (Tao He)
    • [af373dac]: fix gremlin-test for mg (#489) (tianliplus)
    • [df2cc029]: Script support local deployment in MacOS (#464) (Weibin Zeng)
    • [266cccae]: Set grpc max message length options in coordinator server side (#478) (DongZe Li)
    • [16c7766c]: fix store bug (#477) (tianliplus)
    • [e35c717a]: Fix string oid graph transformation bug (#467) (Weibin Zeng)
    • [c3203acf]: fix gaia server lifecycle (#474) (tianliplus)
    • [5508e6b8]: Upgrade vineyard to v0.2.5 (#466) (Tao He)
    • [e0395fb0]: Polymorphic ContextDAGNode to guide selector for user (#461) (DongZe Li)
    • [ba47539b]: Gaia server async start (#463) (tianliplus)
    • [42575b1f]: Fix bug in directed graph transformation between graphscope and nx (#454) (Weibin Zeng)
    • [7bb5a8de]: Add instructions for mount volume in mac (#457) (Siyuan Zhang)
    • [809842a6]: integrate gaia to persistent store (#440) (tianliplus)
    • [6eecccd7]: add mininum properties in graph/vertex step for new graph store api (#455) (shirly121)
    • [d4eac09a]: Add pagerank in NetworkX version (#435) (Weibin Zeng)
    • [e76c81d1]: Support with keyword in session (#451) (Siyuan Zhang)
    • [9e99de7b]: add support of property name to id in logic plan (#446) (shirly121)
    • [3d423379]: [gremlin_core] remove get_partition_num() in Partitioner and refine s… (#449) (BingqingLyu)
    • [ab027368]: Fix building config for graphscope-jupyter extensions (#450) (Moyee)
    • [aa7f81d8]: add interface (#448) (tianliplus)
    • [fd536d42]: add SimpleServerDetector (#438) (tianliplus)
    • [0ad8b428]: Refine gen source (#427) (BingqingLyu)
    • [675d6ad0]: [graphscope] add mapping of partition id to server id in GraphPartitionManager (#431) (BingqingLyu)
    • [85ecfe9b]: return address for jna (#433) (tianliplus)
    • [efb0728b]: Add unload graph/app test cases for lazy mode execution (#428) (DongZe Li)
    • [f3a4850a]: Set locale to utf8 (#425) (Siyuan Zhang)
    • [90099840]: Local CI: use latest vineyard (v0.2.4). (#423) (Tao He)
    • [8690da05]: refactor jna (#419) (tianliplus)
    • [6a24fa3e]: Support prop_id in GAIA runtime (#421) (BingqingLyu)
    • [959c38fc]: fix gremlin test case (#416) (tianliplus)
    • [a92ebe98]: refine bulk load (#413) (tianliplus)
    • [40b744f1]: Add k-hop for example and benchmark (#407) (bmmcq)
    • [56868050]: Use app-version as default tag instead of latest in graphscope-store charts (#406) (Siyuan Zhang)
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Jun 11, 2021)

    Today, we’re announcing the availability of GraphScope 0.5.0.

    This release including a persistent graph store to enable a “service mode” for real-time graph computation, and lazy evaluation of GraphScope programs, an execution strategy that delays the execution of a GraphScope program until needed for efficiency. Also, we provide a Q&A page for beginners.

    GraphScope-Store: A persistent store for mutable graphs

    • A bulk-load tool to import a property graph from files to the GraphScope-Store instance
    • Helm support to launch GraphScope as service with store
    • Support for multiple session connections to a store
    • Interactive queries on graphs in the store with Gremlin

    Enhanced NetworkX compatibility

    • Adaption on the engine to support multi-queries on duplications of the whole graph
    • Based on aforementioned adaption, support all-pairs shortest paths length and closeness centrality algorithms

    Commits

    • [29b8701b]: [networkx] Adapt the networkx import to eager/lazy session mode (#377) (Weibin Zeng)
    • [131fea73]: Check version compatibility during session connecting (#403) (DongZe Li)
    • [5f1b5b0c]: Refine graphscope-store charts (#402) (Siyuan Zhang)
    • [3fa11314]: Add doc for data loader (#397) (tianliplus)
    • [e4d0ebda]: Add Lazy mode support to GraphScope (#340) (DongZe Li)
    • [65a0f4ed]: Ensure graphscope.proto preponderate over outside proto directory. (#401) (Tao He)
    • [5e9a7075]: Test release step (#394) (Siyuan Zhang)
    • [6adec620]: Suggest users to upgrade their pip. (#396) (Tao He)
    • [9ae7b540]: Ignore store service test in k8s test. (#389) (Siyuan Zhang)
    • [8001f562]: Fix for type number changes. (#388) (Siyuan Zhang)
    • [b8a28814]: Persistent storage add DDL supports && python API (#370) (tianliplus)
    • [1988e78a]: Add google analytical links to docs. (Tao He)
    • [28bffc64]: Hotfix for misspelled property name (#384) (Siyuan Zhang)
    • [e10ff69e]: Use codecov v1.0.5. (Tao He)
    • [471ec74f]: Add all pair shortest path length app and implement Dynamic context for complicated result (#363) (Weibin Zeng)
    • [8100c3be]: fix: Correctly Visualization in Jupyterlab Dark theme (#375) (Moyee)
    • [bf8662d0]: Separate test_ctx_builtin.py to avoid oom_ (#371) (Weibin Zeng)
    • [336ec947]: [GAIA] add encode for single event (#369) (yyy)
    • [6a98c7e5]: Add closeness_centrality app and test case of duplicated mode modification (#336) (Weibin Zeng)
    • [909e98fb]: Update README.md (Jingbo Xu)
    • [0b7e3147]: Upgrade vineyard to v0.2.2. (#366) (Tao He)
    • [0525b0ee]: Support "preemptive" params in GraphScope charts (#364) (DongZe Li)
    • [28d4b3bc]: Add persistent storage (#361) (tianliplus)
    • [0226e00c]: Update badges on README.md (#359) (Jingbo Xu)
    • [3e55fce6]: Support duplicated mode in DynamicFragment (#325) (Weibin Zeng)
    • [ec485715]: [GAIA] Redo scope tag implement, to reduce runtime overhead; (#358) (bmmcq)
    • [dcc60a18]: Add FAQ doc (#346) (DongZe Li)
    • [c7ca8155]: [GAIA] Bug-fix: fix the nested tasks running in iteration; (#356) (bmmcq)
    • [5574f1ff]: Fix predicate (#354) (BingqingLyu)
    • [acfd4b14]: update early stop test to two workers (#350) (yyy)
    • [1536f8bb]: [GAIA] fix early stop bug in multi-cancel scenario (#349) (yyy)
    • [80d197de]: Move boost download link from bintray to jforg. (Tao He)
    • [18f3f0b2]: [GAIA] update benchmark for typical queries and inconsistencies fix (#339) (yyy)
    • [0069bb92]: [BUG-Fix] Fix premature abnormal end of flatmap in subtask; (#341) (bmmcq)
    • [b424da70]: add edge global id (#334) (shirly121)
    • [1e6427e2]: It looks that some github runners has already installed clang-format, we just replace the original one. (Tao He)
    • [87a3cbb7]: Try to rm file generated by rust-protobuf (#335) (DongZe Li)
    • [cf5c743d]: Reduce the minimum resources requirement (#332) (DongZe Li)
    • [23af8d3e]: Reduce redundant copies of session parameters (#330) (DongZe Li)
    • [a276d920]: Fix the tutorial links of document (#331) (Weibin Zeng)
    • [b4854089]: update graphscope jupyter module name to @graphscope/graphscope-jupyter (lidongze0629)
    • [eaee904e]: add gremlin test framework && fix age property type issue (#315) (shirly121)
    Source code(tar.gz)
    Source code(zip)
  • v0.4.1(May 16, 2021)

    We highlight the following fixes and improvements included in this releases:

    • Fix compile error under gcc4.9
    • Fix the bug of early stop optimization of GAIA
    • Correctly behavior of Graph.clear()
    • Fix install graphscope-jupyter extensions and correct the visualization behavior
    • Generalize the session interface to make it extensible
    • Publish graphscope-jupyter npm package for jupyterlab extension
    jupyter labextension install @graphscope/graphscope-jupyter
    

    Commits

    • [391a96cd]: Defaults to source when protocol not support vineyard io stream (#311) (DongZe Li)
    • [40a82296]: Correct the visualization behavior (#310) (DongZe Li)
    • [436c1514]: Drop unnecessary rustfmt.toml (#309) (yyy)
    • [c7fc41f8]: Fix install graphscope-jupyter extensions (#307) (DongZe Li)
    • [7a11d47c]: Install hadoop into the base image for HDFS accessing. (#306) (Tao He)
    • [129c3013]: [GAIA] fix the bug of early stop optimization (#304) (yyy)
    • [a6d06290]: Test GAIA in CI. (#305) (Tao He)
    • [59cbe697]: update networkx zh-cn tutorial and fix Graph.clear() bug (#296) (Weibin Zeng)
    • [d25509f4]: Generalize the session interface to make it extensible (#277) (DongZe Li)
    • [72d83f94]: add zh-cn version of networkx tutorial (#285) (Weibin Zeng)
    • [f9db7d24]: fix compile error under gcc4.9 (#276) (Siyuan Zhang)
    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(Apr 30, 2021)

    Today, we’re announcing the availability of GraphScope v0.4.0. This release focuses on the compatibility improvement with NetworkX, with the aim of allowing users to develop graph applications on large-scale graphs in a distributed environment just like doing this on a single machine. In addition, this release improves the experience of standalone deployment.

    We highlight the following improvements included in this release:

    • Improved compatibility with NetworkX
    • Support Graph/DiGraph types in NetworkX
    • Support NetworkX APIs of operations and manipulations over the above two types of graphs
    • A new tutorial in PlayGround to introduce graph processing with NetworkX APIs.

    Commits

    • [76c0d8f]: [networkx] Remove experimental flag of the networkx module (#273) (Weibin Zeng)
    • [72e79dd]: [networkx] improve the document of networkx style api (#263) (Weibin Zeng)
    • [f1c87e3]: [networkx] refact clear api and implment number_of_selfloops api (#266) (Weibin Zeng)
    • [ed97d6c]: [GAIA] New release of GAIA (#272) (yyy)
    • [f11a596]: Generalize the Kubernetes deployment interface to make it extensible (#269) (DongZe Li)
    • [f7908c1]: Enhancement of build.sh (#271) (Siyuan Zhang)
    • [62b3bbf]: [networkx] add tutorial of NetworkX-style api use (#262) (Weibin Zeng)
    • [bcaa815]: Use IOFactory to create adaptors (#268) (Siyuan Zhang)
    • [67a18a7]: [networkx] implement deepcopy subgraph/edge_subgraph in engine, not return a view (#258) (Weibin Zeng)
    • [31922c1]: [networkx] implement graph views in engine side (#256) (Weibin Zeng)
    • [b08ea46]: Correct author info in coordinator pypi package (#260) (Siyuan Zhang)
    • [8ca1aca]: [GAIA] dead lock bug fix of GAIA (#257) (yyy)
    • [e25f983]: [networkx] allowing tuple/bool/double node type (#249) (Weibin Zeng)
    • [9354a24]: [networkx] implement clear_edges, deepcopy of to_directed and to_undirected api (#255) (Weibin Zeng)
    • [1a6b040]: Support launch multiple session/instance in standalone mode (#247) (DongZe Li)
    • [eecec85]: Support graphs that only have vertices or empty. (#245) (Siyuan Zhang)
    • [2b4e891]: update pegasus readme (#250) (yyy)
    • [f1f21f0]: update GAIA: 1. optimize minimal carried properties; 2. add more tests and benchmarks; 3. formatted display of results. (#248) (yyy)
    • [41d1b78]: Support combine multiple add vertice and edges operation. (#241) (Siyuan Zhang)
    • [d858a36]: Add makefile target to simplify process of compilation (#240) (DongZe Li)
    • [0183a32]: update readme (#239) (Siyuan Zhang)
    • [a12ed52]: Don't link "mpi_cxx" on Mac, since brew doesn't install that. (#238) (Tao He)
    • [6c8514b]: Fix several bugs of project/add column listed by #215 (#234) (Siyuan Zhang)
    • [90095c5]: Add template for frontend (#233) (Siyuan Zhang)
    • [be9e82f]: update tutorials and helm readme (#232) (DongZe Li)
    • [11b5fab]: Fix default session for processing vineyard operation. (#230) (Siyuan Zhang)
    • [c7c27c7]: enrich unittest of project and add_column (#215) (Weibin Zeng)
    • [93d92ea]: CI: fixes computing tag when pushing docs to gh-pages branch. (Tao He)
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Apr 3, 2021)

    Following GraphScope v0.2.1 pre-release, we highlight the following improvements included in this release:

    • Integration with mars
    • Graph Visualization: a graphscope-jupyter extension for jupyterlab is experimentally available.
    • Better getting started experience for end-users: Support to deploy GraphScope with standalone of Ubuntu 20
    • Improve graph manipulation APIs with a general project operator
    • Support Louvain algorithm in built-in app.

    Released Images

    sudo docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.3.0
    sudo docker pull registry.cn-hongkong.aliyuncs.com/graphscope/maxgraph_standalone_manager:0.3.0
    

    Commits

    • [9bd97a4]: Fixes mars version to 35b44ed56e031c252e50373b88b85bd9f454332e. (#228) (Tao He)
    • [03a7b03]: New version of GAIA for research: 1. minimum carried information for gremlin traversal; 2. add benchmark and tests; 3. engine service optimization and refactor; 4. support distributed execution. (#229) (yyy)
    • [560309e]: add scripts to install dependencies and build GraphScope (#222) (Weibin Zeng)
    • [1c8739e]: Hotfix for #224. (#227) (Siyuan Zhang)
    • [219ca56]: Use None to select all property, and empty list to deselect all. (#224) (Siyuan Zhang)
    • [af4f6e6]: Standalone support for GraphScope with ubuntu-20 (#214) (DongZe Li)
    • [464b0a4]: Link against mpi_cxx as well. (#223) (Tao He)
    • [405e91c]: Rename to graphscope-jupyter. (#217) (Tao He)
    • [ea17259]: Add a general project operator. (#211) (Siyuan Zhang)
    • [4037c18]: [fix] Test_with_mars not found data_dir (#213) (Weibin Zeng)
    • [b384563]: Support launching mars along with GraphScope engines. (#210) (Tao He)
    • [11df85b]: Hide versions and show languages by default. (#212) (Tao He)
    • [4853cc3]: Add helm test (#209) (DongZe Li)
    • [b47f1e6]: Add a parameter to helm to use prelaunched vineyar DaemonSet. (#204) (Tao He)
    • [d64742c]: Docs requires doxygen. (#203) (Tao He)
    • [fd2481c]: Multi-version docs. (#201) (Tao He)
    • [c357855]: Redesign the way to initialize a graph and revert remove (#199) (Siyuan Zhang)
    • [4f5db65]: Implement louvain algorithm (#183) (Weibin Zeng)
    • [509b915]: support dangling handle for each session connection (#198) (DongZe Li)
    • [b8cb7c4]: Distributed ETCD with 3 pods. (#197) (DongZe Li)
    • [d180191]: Rewrite graph loading API. (#193) (Siyuan Zhang)
    • [7a8f3f6]: Some fixes for running with helm deployed (#194) (DongZe Li)
    • [b76b193]: Add preemptive param in session to give pod a good chance of being sc… (#192) (DongZe Li)
    Source code(tar.gz)
    Source code(zip)
  • v0.2.1(Mar 15, 2021)

    We highlight the following improvements included in this releases:

    • Graph Visualization: Add a module of jupyter notebook extension to visualize the graph data.
    • Better getting started experience for end-users: Optimate loading process of graph and improve the response time of each step in python
    • Better Kubernetes integration: Support deploy with HELM and generalize k8s_volumes to support mount PVC.
    • Improve graph manipulation APIs and support loading from vineyard stream by using names.
    • Support add vertices and edges to existing graph
    • Provide a script to launch k8s cluster on amazon/aliyun cloud.

    Some Breaking API Changes:

    • Deprecate the support for minikube (k8s_minikube_vm_driver)

    Commits

    • [f807eef]: Stop propgagating signal to coordinator subprocess when launching locally (#190) (DongZe Li)
    • [6f98f82]: Refine Graph.project_to_simple method. (#186) (Siyuan Zhang)
    • [d4bd566]: Fixes the typo in docs for pagerank. (#188) (Tao He)
    • [6c0efef]: Helm integration of GraphScope (#184) (DongZe Li)
    • [a232871]: Add vertices and edges to existing graph. (#179) (Siyuan Zhang)
    • [6a0e5a3]: Fix bug about can not join current thread during graph unloading (#177) (DongZe Li)
    • [e5b5e09]: fix the requirements check in launch_cluster script (#176) (Weibin Zeng)
    • [d709119]: Fix issue of union result parser in the new compiler of GIE (#175) (shirly121)
    • [be74042]: Provisioning k8s cluster on amazon/aliyun with script (#168) (Weibin Zeng)
    • [b3cf8f6]: Optimize time of sess.gremlin and sess.close (#170) (DongZe Li)
    • [5980176]: Opensource code for GAIA (#173) (yyy)
    • [705a7e6]: Init gaia directory (#171) (yyy)
    • [10f36b1]: The log_level could be in lower case, transform it to upper first. (#169) (Tao He)
    • [5b5061d]: Print useful information for commonly used objects. (#167) (Siyuan Zhang)
    • [70bb6e5]: Support loading from vineyard stream/dataframe using names. (#166) (Tao He)
    • [d0251fa]: Pre-compile a set of graphs / apps in docker-build stage. (#163) (Siyuan Zhang)
    • [f643d9f]: Capture KeyboardInterrupt during the session launching and cleanup re… (#162) (DongZe Li)
    • [ef3df2c]: Put config to handle in get_handle (#159) (Weibin Zeng)
    • [954f8a6]: Upgrade graph-learn submodule. (#158) (Tao He)
    • [b7ed6d8]: Generalize k8s_volumes and support mount pvc (#156) (DongZe Li)
    • [055480b]: Expose graph interactive engine parameters to client. (#155) (Siyuan Zhang)
    • [0c56ff0]: Optimize source chain operator (#152) (Siyuan Zhang)
    • [16a93ea]: fix: update graphscope package version & readme file (#148) (Moyee)
    • [567e3dc]: Unload graph when session close (#146) (DongZe Li)
    • [3081a8a]: Visualization with ipygraphin jupyterlab extension (#127) (DongZe Li)
    • [f97e26b]: Deprecated 'k8s_minikube_vm_driver' parameter in session. (#129) (DongZe Li)
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Feb 2, 2021)

    The GraphScope team is pleased to announce the 0.2.0 release after two-months development. The 0.2.0 release is focused on better getting started experience for end-users and we have make a lot of improvements in since the last minor release.

    We highlights the following improvements included in this releases:

    1. Better Kubernetes integration: supported LoadBalancer service type and a volumes parameter for graphscope.session() to mount users data into worker pods.
    2. Enhancement for interactive query: added a InteractiveQuery.traversal_source() method for better gremlin experience.
    3. Better I/O and serialization support: started to support various I/O and graph serialization/deserialization.

    We thanks everyone in the graphscope team as well as the community for make graphscope better. We highlight the following commits that involved in this release:

    Commits

    • [f0abbbf5]: add tutorials. (#117) (Jingbo Xu)
    • [33b7e975]: fix compile app with string oid graph, add a simple test (#120) (Siyuan Zhang)
    • [3bc97d93]: Add module string. (#121) (Siyuan Zhang)
    • [2693c128]: Fixes the global property graph stream for subgraph. (#118) (Tao He)
    • [30022c9d]: unify the local k8s cluster setup tool with kind (#116) (Weibin Zeng)
    • [512a5b8a]: Requires vineyard-io, and drop the copy hack. (#115) (Tao He)
    • [c63c25fd]: serialize and deserialize (#103) (Siyuan Zhang)
    • [adb15111, 995f2237]: Update README.md (Wenyuan Yu)
    • [b421e880]: Fix passing zookeeper memory to etcd (#111) (DongZe Li)
    • [6865c1b5]: Check os compatibility and dependencies version in prepare_env.sh (#99) (Weibin Zeng)
    • [e2854d10]: Change session status to disconnected when grpc failed to coordinator (#105) (DongZe Li)
    • [d16ff635]: enable create session from an endpoint (#101) (Weibin Zeng)
    • [3c298899]: Make GraphScope support ResourceQuota limit in namespace (#102) (DongZe Li)
    • [2250fc47]: Parameterized k8s volume mount in session config (#100) (DongZe Li)
    • [1d2a5417]: Fix loading from streams when there's worker that receives no chunks (#98) (Tao He)
    • [5699613f]: Fix bug: delete permissions after deleting all resources. (#97) (DongZe Li)
    • [2695b653]: Try to read namespace from kube config when missing k8s_namesapce param (#95) (DongZe Li)
    • [aeca19db]: Add I/O support for aws s3. (#91) (Siyuan Zhang)
    • [f67e7e20]: Update README-zh.md (#87) (Jingbo Xu)
    • [cc16d6a9]: remove useless dependency (#84) (shirly121)
    • [99d6e46c]: Update badges for docs. (#85) (Jingbo Xu)
    • [ac4c1e2b]: Upgrade vineyard to v0.1.5. (#73) (Tao He)
    • [9e9d37e7]: add app kshell (#66) (Yongmin Hu)
    • [78404adc]: Update badge in docs. (#79) (Jingbo Xu)
    • [924af133]: Update graphlearn submodule for fixing the closing client issue (#77) (Weibin Zeng)
    • [53636b23]: Add Chinese docs for GraphScope (#67) (Jingbo Xu)
    • [dcd081de]: Fix cross-reference links in GIE docs (#74) (yyy)
    • [0915a56f]: [BUGFIX] Expose the correct external IP of GIE. (#69) (DongZe Li)
    • [26d256f4]: Initial version of the Chinese docs for GIE (#68) (yyy)
    • [97f41adf]: Render zh_CN docs under docs/zh/ directory, and add the language selector. (#61) (Tao He)
    • [5c9a2224]: Fixed the version at the time of releasing the pip package. (#49) (DongZe Li)
    • [39e39850]: Extend gremlin pod expose to load-balancer (#45) (shirly121)
    • [12de77c0]: Move session params show_log and log_level as a global configuration. (#42) (DongZe Li)
    • [595d1c7e]: combine prepare_env scripts into one (#38) (Weibin Zeng)
    • [46ab90a9]: Enforce vineyard version to find. (#39) (Tao He)
    • [e21d946b]: Support bytecode-based requests for interactive engine (#36) (Siyuan Zhang)
    • [b46a0683]: Add: provide configurable parameters for connecting to remote k8s.(#37) (#40) (howie.hu)
    • [dd2a58ac]: Fixes license info in pypi package (setup.py). (#34) (Tao He)
    • [7725f766]: Fix release image step: checkout code first before building and pushing image (#31) (Siyuan Zhang)
    • [8fd52b4e]: Migrate CI to GitHub runners. (#21) (Siyuan Zhang)
    • [d28c6631]: Wait etcd ready before launching vineyard container. (#23) (Tao He)
    • [69f401ed]: Add load balance service type (#24) (DongZe Li)
    • [edc0b0ed]: check docker is running or not in prepare_env.sh/prepare_env_wsl.sh (#17) (Weibin Zeng)
    Source code(tar.gz)
    Source code(zip)
Owner
Alibaba
Alibaba Open Source
Alibaba
A graph library for Rust.

Gamma A graph library for Rust. Gamma provides primitives and traversals for working with graphs. It is based on ideas presented in A Minimal Graph AP

Metamolecular, LLC 121 Nov 19, 2022
Simple but powerful graph library for Rust

Graphlib Graphlib is a simple and powerful Rust graph library. This library attempts to provide a generic api for building, mutating and iterating ove

Purple Protocol 177 Nov 22, 2022
🦀 Rust Graph Routing runtime for Apollo Federation 🚀

Apollo Router The Apollo Router is a configurable, high-performance graph router for a federated graph. Getting started Follow the quickstart tutorial

Apollo GraphQL 468 Nov 25, 2022
A simple and elegant, pipewire graph editor

pw-viz A simple and elegant, pipewire graph editor This is still a WIP, node layouting is kinda jank at the moment. Installation A compiled binary is

null 178 Nov 21, 2022
A graph crate with simplicity in mind

A graph crate with simplicity in mind. Prepona aims to be simple to use (for users of the crate) and develop further (for contributors). Nearly every

Mohamad Amin Rayej 75 Nov 22, 2022
Graph API client writen in Rust

graph-rs Now available on stable Rust at crates.io graph-rs-sdk = "0.1.0" 0.1.0 and above use stable Rust. Anything before 0.1.0 uses nightly Rust. M

Sean Reeise 52 Nov 21, 2022
Rust library for of graph ensembles

Rust library for random graph ensembles Minimal Rust version: 1.55.0 Implements simple sampling and monte carlo (or rather markov-) steps, that can be

Yannick Feld 2 Nov 16, 2022
A Graph implemented using nothing but `Vec`s in rust

VecGraph A Graph implemented using nothing but Vecs in rust. Details The graph is implemented using two Vecs: nodes and edges. nodes stores "nodes". w

null 1 Jan 27, 2022
Simple, performant graph generator for Feynman diagrams* ⚛️

Feynman diagram generator ⚛️ A simple generator of "Feynman diagram" permutations (as defined by problem 781). Incrementally builds isomorphically uni

eugene huang 2 Sep 7, 2022
3D Solar System visualization

Description This project is an unrealistic 3D Solar System visualization. It was made for Computer graphics course, that I took at the University of W

Michał Sala 1 Feb 22, 2022
Hypergraph rewriting system with avatars for symbolic distinction

Avatar Hypergraph Rewriting Hypergraph rewriting system with avatars for symbolic distinction Based on paper Avatar Hypergraph Rewriting. Avatars can

AdvancedResearch 2 Nov 14, 2021
rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.

rbdt ?? ?? ?? ?? rbdt is a work in progress, currently being extracted out of another (private) project for the purpose of open sourcing and better so

Knuckleheads' Club 0 Nov 9, 2021
Codemod - Codemod is a tool/library to assist you with large-scale codebase refactors that can be partially automated but still require human oversight and occasional intervention

Codemod - Codemod is a tool/library to assist you with large-scale codebase refactors that can be partially automated but still require human oversight and occasional intervention. Codemod was developed at Facebook and released as open source.

Meta Archive 4k Dec 1, 2022
Thalo is an event-sourcing framework for building large scale systems

Thalo Event sourcing framework for building microservices. Overview Thalo is an event-sourcing framework for building large scale systems based on the

null 530 Dec 4, 2022
Decode SCALE bytes into custom types using a scale-info type registry and a custom Visitor impl.

scale-decode This crate attempts to simplify the process of decoding SCALE encoded bytes into a custom data structure given a type registry (from scal

Parity Technologies 6 Sep 20, 2022
One-Stop Solution for all boilerplate needs!

One Stop Solution for all boilerplate needs! Consider leaving a ⭐ if you found the project helpful. Templa-rs Templa-rs is a one-of-a-kind TUI tool wr

IEEE VIT Student Chapter 27 Aug 28, 2022
Your one stop CLI for ONNX model analysis.

Your one stop CLI for ONNX model analysis. Featuring graph visualization, FLOP counts, memory metrics and more! ⚡️ Quick start First, download and ins

Christopher Fleetwood 19 Nov 19, 2022
Hooks for Yew, inspired by streamich/react-use and alibaba/hooks.

Yew Hooks Hooks for Yew, inspired by streamich/react-use and alibaba/hooks. Hooks State use_toggle - tracks state of counterparts. use_bool_toggle - t

Jet Li 107 Nov 24, 2022
Egui node graph is a featureful, customizable library to create node graph applications using egui

Egui node graph is a featureful, customizable library to create node graph applications using egui. The library takes care of presenting a node graph to your users, and allows customizing many aspects of the interaction, creating the semantics you want for your specific application.

null 329 Dec 2, 2022
Common stop words in a variety of languages

About Stop words are words that don't carry much meaning, and are typically removed as a preprocessing step before text analysis or natural language p

Chris McComb 8 Oct 7, 2022