Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support docarray v2 #5603

Merged
merged 76 commits into from Jan 27, 2023
Merged

feat: support docarray v2 #5603

merged 76 commits into from Jan 27, 2023

Conversation

samsja
Copy link
Contributor

@samsja samsja commented Jan 16, 2023

Context

this PR is a PoC to support docarray v2 in jina.

This pr make jina compatible with v2 while keeping v1 support as well.

PS: Some of the features are yet not compatible with v2, like filtering etc, but this is an entry point for letting people play with v2.

what this pr do

allow to do

from jina import Flow, Executor, requests
from docarray import BaseDocument, DocumentArray
from docarray.documents import Image
from docarray.typing import AnyTensor

class InputDoc(BaseDocument):
    img: Image

class OutputDoc(BaseDocument):
    embedding: AnyTensor

class MyExec(Executor):
    @requests(on='/bar')
    def bar(
        self, docs: DocumentArray[InputDoc], **kwargs
    ) -> DocumentArray[OutputDoc]:

        docs_return = DocumentArray[OutputDoc](
            [OutputDoc(embedding=np.zeros((100, 1))) for _ in range(len(docs))]
        )

        return docs_return

with Flow().add(uses=MyExec) as f:
    docs = f.post(
        on='/bar',
        inputs=InputDoc(img=Image(tensor=np.zeros((3, 224, 224)))),
        return_type=DocumentArray[OutputDoc],
    )
    assert docs[0].embedding.shape == (100, 1)
    assert docs.__class__.document_type == OutputDoc

FastAPI:

import pytest

from fastapi import FastAPI
from httpx import AsyncClient

from docarray import BaseDocument
from docarray.base_document import DocumentResponse
from docarray.documents import Image
from docarray.typing import AnyTensor


import numpy as np


class InputDoc(BaseDocument):
    img: Image


class OutputDoc(BaseDocument):
    embedding: AnyTensor


input_doc = InputDoc(img=Image(tensor=np.zeros((3, 224, 224))))

app = FastAPI()


@app.post("/embed/", response_model=OutputDoc, response_class=DocumentResponse)
async def embed(doc: InputDoc) -> OutputDoc:
    doc = OutputDoc(embedding=np.zeros((100, 1)))
    return doc


async with AsyncClient(app=app, base_url="http://test") as ac:
    response = await ac.post("/embed/", data=input_doc.json())

doc = OutputDoc.parse_raw(response.content.decode())
assert doc.embedding.shape == (100, 1)

Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>

wip

Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>

feat: compatible with docarray b2

Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja samsja requested a review from hanxiao as a code owner January 16, 2023 13:56
@github-actions github-actions bot added size/M area/core This issue/PR affects the core codebase area/entrypoint This issue/PR affects the entrypoint codebase area/helper This issue/PR affects the helper functionality area/network This issue/PR affects network functionality component/client component/proto component/type labels Jan 16, 2023
@JoanFM JoanFM marked this pull request as draft January 16, 2023 15:03
@github-actions github-actions bot added the area/setup This issue/PR affects setting up Jina label Jan 19, 2023
@github-actions github-actions bot removed component/proto area/network This issue/PR affects network functionality labels Jan 19, 2023
@github-actions github-actions bot added area/network This issue/PR affects network functionality component/proto labels Jan 19, 2023
@github-actions github-actions bot added the area/testing This issue/PR affects testing label Jan 19, 2023
samsja and others added 3 commits January 26, 2023 12:20
Co-authored-by: Joan Fontanals <joan.martinez@jina.ai>
Co-authored-by: Joan Fontanals <joan.martinez@jina.ai>
Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
samsja and others added 3 commits January 26, 2023 13:02
Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Show resolved Hide resolved
tests/integration/docarray_v2/test_v2.py Show resolved Hide resolved
tests/integration/docarray_v2/test_v2.py Show resolved Hide resolved
Co-authored-by: AlaeddineAbdessalem <alaeddine-13@live.fr>
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
samsja and others added 3 commits January 26, 2023 14:12
Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com>
Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
CHANGELOG.md Outdated Show resolved Hide resolved
Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
JoanFM
JoanFM previously approved these changes Jan 27, 2023
Copy link
Member

@alexcg1 alexcg1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few things I missed on the first round

docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved

Jina provides early support for [DocArray v2](https://github.com/docarray/docarray/commits/feat-rewrite-v2) which
is a rewrite of DocArray. DocArray v2 makes the dataclass feature of DocArray-v1 a first-class citizen and for this
purpose it is built on top of [pydantic](https://pydantic-docs.helpmanual.io/) and . An important shift is that
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and...what?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or was this an editing glitch on my part from earlier?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

euh yeah I think so ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay actually it is on the next line nothing wrong

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uh, I still see the dangling and .

image

docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
docs/concepts/executor/docarray-v2.md Outdated Show resolved Hide resolved
Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com>
@github-actions
Copy link

📝 Docs are deployed on https://feat-docarray-v2--jina-docs.netlify.app 🎉

Copy link
Member

@alexcg1 alexcg1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/cicd This issue/PR affects the cicd pipeline area/core This issue/PR affects the core codebase area/docs This issue/PR affects the docs area/entrypoint This issue/PR affects the entrypoint codebase area/helper This issue/PR affects the helper functionality area/housekeeping This issue/PR is housekeeping area/network This issue/PR affects network functionality area/testing This issue/PR affects testing component/client component/jaml component/proto component/type size/L size/M
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants