Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Sonic Channel libraries for popular programming languages #69

Closed
7 of 8 tasks
valeriansaliou opened this issue Mar 23, 2019 · 27 comments
Closed
7 of 8 tasks
Labels
help wanted Extra attention is needed

Comments

@valeriansaliou
Copy link
Owner

valeriansaliou commented Mar 23, 2019

Sonic cannot be integrated to an existing application code without a Sonic Channel library made for the used programming language.

Currently, a NodeJS library is available node-sonic-channel, but that's all!

We'd need to support more programming languages with clean implementations of the Sonic Channel protocol as specified in: PROTOCOL.

Programming languages:

  • Rust
  • JavaScript
  • Go
  • Java
  • Python
  • Ruby
  • PHP
  • (any other language?)
@valeriansaliou
Copy link
Owner Author

@moshe is working on a Python binding, see #82 and https://github.com/moshe/asonic

@alexisvisco
Copy link

alexisvisco commented Mar 25, 2019

Hello @valeriansaliou I have implemented a tiny go library for sonic https://github.com/expectedsh/go-sonic
Thanks you for your project !

@valeriansaliou
Copy link
Owner Author

Thank you so much, I'm adding it to the README right now.

@asonix
Copy link

asonix commented Mar 25, 2019

Is there a Rust client library provided already? I imagine since the server is written in rust, it wouldn't be too difficult to put together a rust client.

@valeriansaliou
Copy link
Owner Author

valeriansaliou commented Mar 25, 2019 via email

@kaspermeyer
Copy link

If no one else has started working on it, I’d love to take on the Ruby client!

@valeriansaliou
Copy link
Owner Author

@kaspermeyer I confirm that to my knowledge no one has yet started working on the Ruby client, you can go for it 😄 Thanks!

@seijikun
Copy link

Hey guys! I wrote a small php client library.
I am planning on using this to write a backend for Nextcloud's fulltextsearch framework.

@valeriansaliou
Copy link
Owner Author

@seijikun thanks! It's been added to the README in commit 777734c 👍

@alexisvisco
Copy link

I have implemented a bulk feature in my library, maybe the official library (in nodejs) should have it ?
https://github.com/expectedsh/go-sonic/blob/master/sonic/ingester.go#L113

@valeriansaliou
Copy link
Owner Author

@alexisvisco thanks, looks neat. I'll think about it for the NodeJS library. At the network level, it's still the same so there's no (speed) optimizations involved w/ a bulk feature in the library, though it reduces the lines of code for users that need it.

@alexisvisco
Copy link

alexisvisco commented Mar 27, 2019

In my implementation I have splitted the number of coroutines and assign a new connection to it (because in Go you can't read a connection by multiple coroutines and it's normal).

For example:
There is an array of 10 bulk values.
I decide to split into 2 coroutines (parallelRoutines field).
There is now 2 arrays of 5 elements that will be executed in parallel.

Each coroutines is independent so there is an optimisation because of the paralelization, you can speed up to 2 times faster in this case (if the sonic server can handle this traffic).
By the way I have executed a benchmark of this and it is positive, look at this 😄

@valeriansaliou
Copy link
Owner Author

valeriansaliou commented Mar 27, 2019

@alexisvisco I see, you're able to increase the concurrency level at the library level. Sonic will definitely scale this better with your model, indeed. Depending on the number of cores of the machine it'll use CPU cores at their best as Sonic spawns an independent thread per Sonic Channel connection.

I have plans to add a worker pool to Sonic to handle the query + suggest commands. I've already made the protocol async-compatible for search mode (in advance), so any "worker pool" improvement in Sonic will not require any change in the protocol.

@valeriansaliou
Copy link
Owner Author

valeriansaliou commented Mar 27, 2019

@alexisvisco just curious, in which context / for which project are you using Sonic, if so? What motivated you to build the Golang integration? (if that can be said publicly)

@c-kirkeby
Copy link

I'll probably have a go at making a native Rust client. Should make for a good learning experience!

@alexisvisco
Copy link

alexisvisco commented Mar 28, 2019

@alexisvisco just curious, in which context / for which project are you using Sonic, if so? What motivated you to build the Golang integration? (if that can be said publicly)

@valeriansaliou Expected.sh is a 🇫🇷 container as a service, we want to build something awesome.

We wanted to use Sonic to ingest logs and be able to search them. However we noticed that it does not really fit because we would have wanted to be able to filter over time, and for now we can't with Sonic. We will keep an eye on sonic but we will rather use something else.

To be honest I have created the Go driver because our stack is in Go and I wanted to quickly know all of sonic. And also because the project looked cool 😎

@atipugin
Copy link

Hi! I've created a Ruby client for Sonic - https://github.com/atipugin/sonic-ruby. There are some improvements to do, but it's already usable :)

@valeriansaliou
Copy link
Owner Author

Thanks @atipugin I've added it to the README 👍

@seijikun
Copy link

@touhonoob and me have merged our libraries into his repository, so you can remove my repo from the README. 😃

@valeriansaliou
Copy link
Owner Author

@seijikun done in cfbd3ac

@alohaking
Copy link

Hi,I found that the current java client is synchronous, so I wrote an asynchronous client using netty.https://github.com/alohaking/jsonic

@valeriansaliou
Copy link
Owner Author

@alohaking can you submit a PR where you add the link to your library in the README, below the other Java library? Thanks!

valeriansaliou added a commit that referenced this issue Apr 1, 2019
Signed-off-by: Valerian Saliou <valerian@valeriansaliou.name>
@valeriansaliou
Copy link
Owner Author

@alohaking just did it myself. Thanks!

@OGKevin
Copy link
Contributor

OGKevin commented Apr 6, 2019

Hmm I see that im late to the party :( https://github.com/OGKevin/go-sonic

#77 (comment)

The only thing is missing is proper client documentation, make sure the query length is checked to prevent overflow and actually implementing it in my own project.

@valeriansaliou
Copy link
Owner Author

You can still submit a PR to add your library to Sonic's README ;)

@FrontMage
Copy link

Hi, I made a Rust client sonic_client, using mio and channels.
There is a question though, is QUERY and SUGGEST share the same id pool? Do I need make 2 maps for them or just one.

@valeriansaliou
Copy link
Owner Author

Hi there! Thanks a ton, adding it to README now!

Regarding your question, yes, they can share the same ID pool as all async events are routed on the same TCP channel (in search mode).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

10 participants