发布新帖

查找

文章
· 五月 15, 2024 阅读大约需 6 分钟

IRIS AI Studio: Connectors to Transform your Files into Vector Embeddings for GenAI Capabilities

In the previous article, we saw different modules in IRIS AI Studio and how it could help explore GenAI capabilities out of IRIS DB seamlessly, even for a non-technical stakeholder. In this article, we will deep dive into "Connectors" module, the one that enables users to seamlessly load data from local or cloud sources (AWS S3, Airtable, Azure Blob) into IRIS DB as vector embeddings, by also configuring embedding settings like model and dimensions. 

 

New Updates  ⛴️ 

  • Online Demo of the application is now available at https://iris-ai-studio.vercel.app
  • Connectors module can now load data from (OpenAI/Cohere embeddings)
    • Local Storage
    • AWS S3
    • Azure Blob Storage
    • Airtable
  • Playground module is fully functional with
    • Semantic Search
    • Chat with Docs
    • Recommendation Engine
      • Cohere Re-rank
      • OpenAI Re-rank
    • Similarity Engine

 

Connectors

If you have used ChatGPT 4 or other LLM services where they get the context and run the intelligence on top of that context, that ideally adds business value over a generic LLM. Simply the intelligence on your data. This module out of box gives a no-code interface to load data from different sources, do embeddings on it and load it to IRIS DB. This connectors module mainly go through 3 steps

  1. Fetching data from different sources
  2. Getting the data embedded using OpenAI/Cohere embedding models
  3. Loading the embeddings and text into IRIS DB

 

Step 1: Fetching data from different source

1. Local Storage - Upload files. I have used Llama Index's SimpleDirectoryReader to load data from files. 

(a limitation of up to 10 files in one go is put to handle the load on tiny server that I used for demo purpose. Can be negated on your own implementation)

# Check for uploaded files
if "files" not in request.files:
  return jsonify({"error": "No files uploaded"}), 400
uploaded_files = request.files.getlist("files")
if len(uploaded_files) > 10:
  return jsonify({"error": "Exceeded maximum file limit (10)"}), 400
temp_paths = []
for uploaded_file in uploaded_files:
  fd, temp_path = tempfile.mkstemp()
  with os.fdopen(fd, "wb") as temp:
    uploaded_file.save(temp)
  temp_paths.append(temp_path)

# Load data from files
documents = SimpleDirectoryReader(input_files=temp_paths).load_data()

 

 2. AWS S3

Input Parameters: Client ID, Client Secret and Bucket Name. You may get the client id and secret from the AWS console - IAM or creating read permissions for your bucket over there.

I have used "s3fs" library to fetch the contents from AWS S3 and Llama Index's SimpleDirectoryReader to load data from the fetched files.

access_key = request.form.get('aws_access_key')
secret = request.form.get('aws_secret')
bucket_name = request.form.get('aws_bucket_name')

if not all([access_key, secret, bucket_name]):
    return jsonify({"error": "Missing required AWS S3 parameters"}), 400
s3_fs = S3FileSystem(key=access_key, secret=secret)
reader = SimpleDirectoryReader(input_dir=bucket_name, fs=s3_fs, recursive=True)
documents = reader.load_data()

 

3. Airtable

Input Parameters: Token (API Key), Base ID and Table ID. API Key can be retrieved from the Airtable's Developer Hub. Base ID and Table ID can be found from the table's URL. The one that starts with "app.." is the base ID and "tbl.." is the Table ID

I have used Airtable Reader from LlamaHub to fetch the contents from Airtable and Llama Index's SimpleDirectoryReader to load data from the fetched files.

airtable_token = request.form.get('airtable_token')
table_id = request.form.get('table_id')
base_id = request.form.get('base_id')

if not all([airtable_token, table_id, base_id]):
        return jsonify({"error": "Missing required Airtable parameters"}), 400
reader = AirtableReader(airtable_token)
documents = reader.load_data(table_id=table_id, base_id=base_id)

 

4. Azure Blob Storage: 

Input Parameters: Container name and Connection string. These information can be retrieved from Azure's AD page. 

I have used AzStorageBlob Reader from LlamaHub to fetch the contents from Azure Storage and Llama Index's SimpleDirectoryReader to load data from the fetched files.

container_name = request.form.get('container_name')
connection_string = request.form.get('connection_string')

if not all([container_name, connection_string]):
    return jsonify({"error": "Missing required Azure Blob Storage parameters"}), 400
loader = AzStorageBlobReader(
    container_name=container_name,
    connection_string=connection_string,
)
documents = loader.load_data()

LlamaHub does contain 500+ connectors ranging from different file types to services. Adding a new connector based on your needs should be pretty straight forward. 

 

Step 2: Getting the data embedded using OpenAI/Cohere embedding models

Embeddings are numerical representations that capture the semantics of text, enabling applications like search and similarity matching. Ideally the objective is, when an user asks a question, its embedding is compared to document embeddings using methods like cosine similarity – higher similarity indicates more relevant content. 

Here, I'm using llama-iris library to store embeddings into IRIS DB. In the IRISVectorStore params

  • Connection String is needed for interaction with DB
    • For trying out in the online demo version, you may not use a locally running instance (localhost).
    • You would need an IRIS instance that runs on AWS/Azure/GCP with 2024.1+ version, since those support vector storage and retrieval.
    • The IRIS Community instance provided by the learning hub seems to be running with 2022.1 version, in which case that cannot be used for exploration purpose.
  • Table Name is the one that will be used to create or update records into
    • The library "llama-iris" appends "data_" to the table name. So, when you are trying to check the data through DB client, append "data_" to the table name. Say, you've named table as "users", you would need to retrieve as "data_users"
  • Embed Dim / Embedding Dimension is the dimension of the Embedding model that the user used
    • Say you've loaded "users" table by using OpenAI embeddings - "text-embedding-3-small" with 1536 dimension. You would be able to load more data into the table, but only with 1536 dimension. Same goes with vector embedding retrievals as well. So, make sure to choose the right model in the initial phases.
CONNECTION_STRING = f"iris://{username}:{password}@{hostname}:{port}/{namespace}"
vector_store = IRISVectorStore.from_params(
    connection_string=CONNECTION_STRING,
    table_name=table_name,
    embed_dim=embedding_dimension
)

Settings.embed_model = set_embedding_model(indexing_type, model_name, api_key)

storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context
)

 

Step 3: Loading the embedding and text into IRIS DB

The above written code covers the indexing and loading of data into IRIS DB. Here is how the data would look like - 

Text - Raw text information that's been extracted from the files we loaded

Node ID - This would be used as a reference when we do retrievals 

Embeddings - The actual numerical representations of the text data

 

These three steps are through which the connectors module mainly work. When it comes to required data, like DB credentials and API Keys - I get it from the user and save it to the browser's local storage (Instance Details) and session storage (API Keys). It gives more modularity to the application for anyone to explore.

 

By bringing together the loading of vector-embedded data from files and the retrieval of content through various channels, IRIS AI Studio enables an intuitive way to explore the Generative AI possibilities that InterSystems IRIS offers - not only for existing customers, but also for new prospects. 

🚀 Vote for this application in Vector Search, GenAI and ML contest, if you find it promising!

If you can think of any potential applications using this implementation, please feel free to share them in the discussion thread.

2 Comments
讨论 (2)0
登录或注册以继续
文章
· 五月 15, 2024 阅读大约需 2 分钟

Retrieve images using vector search (2)

You need to install the application first. If not installed, please refer to the previous article

Application demonstration

After successfully running the iris image vector search application, some data needs to be stored to support image retrieval as it is not initialized in the library.

Image storage

Firstly, drag and drop the image or click the upload icon, select the image, and click the upload button to upload and vectorize it. This process may be a bit slow.

This process involves using embedded Python to call the CLIP model and vectorize the image into 512 dimensional vector data.

ClassMethod LoadClip(imagePath) As %String [ Language = python ]
{
import torch
import clip
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

image = preprocess(Image.open(imagePath)).unsqueeze(0).to(device)

with torch.no_grad():
    image_features = model.encode_image(image)
return str(image_features[0].tolist())
}

View vector data

Viewing Vector Data in the Management Portal

Vector query

The image vector query process uses VECTOR_COSINE to retrieve the similarity of vector data, which can be achieved through the following SQL.

select top ? id Image,VECTOR_COSINE(TO_VECTOR(ImageVector,double),(select TO_VECTOR(ImageVector,double) from VectorSearch_DB.ImageDB where id= ? )) Similarity from VectorSearch_DB.ImageDB order by Similarity desc

 

After uploading multiple images, select query and the image with the highest similarity to the left will be displayed on the right. Click next to view the next page. The similarity between the two images will gradually decrease with each click.

 

This application is a demo of image vector retrieval. If you are interested, you can check Open Exchange or visit Github to see more information.

3 Comments
讨论 (3)2
登录或注册以继续
文章
· 五月 15, 2024 阅读大约需 4 分钟

Vector search + GenAI + InterSystems technologies with Banksia Global

 

Hi all! Here I would like to share how we use vector search and GenAI with InterSystems technology. As an example, I'll describe BG-AppealAI project, which our company submitted to the InterSystems Vector Search, GenAI and ML Contest. BG-AppealAI application can write an appeal if you upload an insurance contract and the insurance company’s letter with a refusal to pay medical expenses. Of course, we are aware that at the moment AI has not reached such a level as to create ready-made legal documents. However, it's already possible to create tools to serve as assistants for specialists or ordinary users. Likewise, our application is an AI assistant that answers the question: "Is an appeal possible?" and creates a draft letter of appeal, if possible. 

Explore our application in action by following this link: https://appealai.cloud.banksia.global/#/welcome

We've also created a video showcasing our app: https://www.youtube.com/embed/0p6FvZpzaaA

How BG-AppealAI works

Insurance contracts and rejection letters vectorization: to be able to better assist with appeals, AI needs a context of the denial. One of the possible ways we can provide a context is by citing relevant parts of insurance contracts. Upon receiving an insurance contract, BG-AppealAI breaks it down into parts, converting each part into a multi-dimensional vector. These vectors are stored in an InterSystems IRIS database, allowing fast and accurate retrieval when needed. 

Vector Search:  when you upload a rejection letter to the app, BG-AppealAI converts it to the vector, this is called the vectorization process.  BG-AppealAI then uses IRIS vector search functionality to compare vectors extracted from the rejection letter with vectors extracted from the relevant insurance contract.

AI-Powered Appeal Generation: having identified potential context for appeal, BG-AppealAI sends a request to the generative AI service to assess the possibility of drafting an appeal. If it is deemed appropriate, AppealAI sends a new request in order to create a draft letter of appeal.

 

What's under the hood of the BG-AppealAI

Traditionally, our application consists of backend and frontend parts, where the frontend is developed on Angular framework and the backend on InterSystems IRIS. The video about our application has already shown the work of the frontend part of the application and a brief overview of the product, here I want to focus on the implementation of Interoperability Production. This is where we can clearly see the application of vector search and generative AI. 

So, there are the following elements in our system:
1) ConversionToVector - process that converts document text into vectors using LangChain, and invokes process GetText and operations SaveVector, SaveDocument for the corresponding actions.

 

2) GetText - process that parses data from a file into a stream, for further text processing and analysis. Here there is opportunity to work with 2 types of extensions: docx and pdf, as the most common ones for documents. 

 

3) VectorSearch - process that performs a vector search and looks for matches between documents. Also, it is in this business process that methods are called to write promts to the AI and analyze rejection letters. 

 

Note, that here we used a language model developed by OpenAI and the LangChain framework. All business processes are implemented using embedded python as it provides easy access to utilization to use LangChain and write promts to OpenAI.
4) AppealIn - service that receives the rejection file and sends it for analysis.
5) DocumentIn - service that receives the insurance contract file and sends it for vectorization.
6) OpenAiOut - operation that makes a request to a generative AI - OpenAI, receives promt, returns a response from GenAI.
7) SaveDocument - operation that stores the document file in the database.
8) SaveVector - operation that stores vectors in the database.
9) SaveAppeal - operation that gets the text of the appeal and stores it in the database.

The interaction of all the above elements is well shown by the visual trace. Visual trace of taking an insurance contract, dividing it into vectors, and storing the vectors in a database:

 

Visual trace of receiving a rejection letter, sending requests to OpenAI, and saving the appeal letter if an appeal is possible:

 

This is an example of a rejection letter:

 

This is an example of an appeal written by BG-AppealAI:

 

We hope that our open source project will be useful to this community and help you implement innovative technologies in your projects. Our team would like to thank InterSystems for the opportunity to work with cutting-edge technologies. We're eager to engage with your thoughts and questions in the comments below. Let's continue pushing the boundaries of what's possible with technology together! Stay tuned for more updates and insights from our team.

If you enjoyed getting to know BG-AppealAI and its features and would like to support us, please take a moment to vote for our app here https://openexchange.intersystems.com/contest/36. Thank you!

3 Comments
讨论 (3)5
登录或注册以继续
问题
· 五月 15, 2024

How to insert data to Third party Database?

hi !

now I want to insert data to third party database directly,  Link Table wizard how to insert  except by sql? 

obj=##class(YBFXJC.LABINFO).%New()
  ;s obj.ID="1"
  obj.HISID="1"
  obj.APPYNO="1"
  obj.HOSPITALID="1"
  obj.BILGDRNAME="1"
  sc=obj.%Save()
  sc_$System.Status.GetErrorText(sc),!

 there is error  <CANNOT SET THIS PROPERTY>zSaveData+3^YBFXJC.TestSaveObj.1

4 Comments
讨论 (4)3
登录或注册以继续
讨论
· 五月 15, 2024

【Coding Puzzle】 "Esthetic Numbers"

Hello,

I would like you to propose this challenge.

It has been created by the CodeWars community here: https://www.codewars.com/kata/6523a71df7666800170a1954/python

I will copy and paste the description:

 

🔡🟢 DESCRIPTION:

A number is Esthetic if, in any base from base2 up to base10, the absolute difference between every pair of its adjacent digits is constantly equal to 1.

num = 441 (base10)
// Adjacent pairs of digits:
// |4, 4|, |4, 1|
// The absolute difference is not constant
// 441 is not Esthetic in base10

441 in base4 = 12321
// Adjacent pairs of digits:
// |1, 2|, |2, 3|, |3, 2|, |2, 1|
// The absolute difference is constant and is equal to 1
// 441 is Esthetic in base4

Given a positive integer num, implement a function that returns an array containing the bases (as integers from 2 up to 10) in which num results to be Esthetic, or an empty array [] if no base makes num Esthetic.

Examples

10 ➞ [2, 3, 8, 10]
// 10 in base2 = 1010
// 10 in base3 = 101
// 10 in base8 = 12
// 10 in base10 = 10

23 ➞ [3, 5, 7, 10]
// 23 in base3 = 212
// 23 in base5 = 43
// 23 in base7 = 32
// 23 in base10 = 23

666 ➞ [8]
// 666 in base8 = 1232

 

The initial Python code is just:

def esthetic(num):
    pass

 

And the Unit Tests are:

import codewars_test as test
from solution import esthetic

@test.describe("Fixed tests")
def fixed_tests():
    @test.it('Basic test Cases')
    def basic_test_cases():
        test.assert_equals(esthetic(10), [2, 3, 8, 10], "Example #1")
        test.assert_equals(esthetic(23), [3, 5, 7, 10], "Example #2")
        test.assert_equals(esthetic(666), [8], "Example #3")
        test.assert_equals(esthetic(13), [5, 6])
        test.assert_equals(esthetic(1), [2, 3, 4, 5, 6, 7, 8, 9, 10])
        test.assert_equals(esthetic(9), [4, 7, 9, 10])
        test.assert_equals(esthetic(74), [])
        test.assert_equals(esthetic(740), [4, 6, 9])
        test.assert_equals(esthetic(928), [])
        test.assert_equals(esthetic(259259), [9])
        test.assert_equals(esthetic(883271), [])
        test.assert_equals(esthetic(1080898), [7])
        test.assert_equals(esthetic(1080899), [])

 

 

 

 

My solution was:

def esthetic(num):
    def to_base(n, base):
        """Convert number n to a given base and return the digits as a list."""
        digits = []
        while n:
            digits.append(n % base)
            n //= base
        return digits[::-1]  # reverse the list to get the correct order

    def is_esthetic_in_base(digits):
        """Check if the given list of digits is esthetic."""
        for i in range(1, len(digits)):
            if abs(digits[i] - digits[i - 1]) != 1:
                return False
        return True

    esthetic_bases = []

    for base in range(2, 11):
        digits = to_base(num, base)
        if is_esthetic_in_base(digits):
            esthetic_bases.append(base)

    return esthetic_bases

 

 

How would you like to solve it using ObjectScript or any other prefered programming language?

4 Comments
讨论 (4)1
登录或注册以继续