Summary Buffer Memory¶

The same CassandraChatMessageHistory can power a more sophisticated kind of chat memory that automatically maintains a summary of the past interactions, so as to provide a memory that virtually extends arbitrarily far back in the past while fitting a finite amount of tokens.

Notice¶

this demo is currently not working with the Azure OpenAI LLMs. Please run it with another LLM provider.

In [1]:

from langchain.memory import CassandraChatMessageHistory
from langchain.memory import ConversationSummaryBufferMemory
from langchain.chains import ConversationChain

from langchain.memory import CassandraChatMessageHistory from langchain.memory import ConversationSummaryBufferMemory from langchain.chains import ConversationChain

In [2]:

from cqlsession import getCQLSession, getCQLKeyspace
cqlMode = 'astra_db' # 'astra_db'/'local'
session = getCQLSession(mode=cqlMode)
keyspace = getCQLKeyspace(mode=cqlMode)

from cqlsession import getCQLSession, getCQLKeyspace cqlMode = 'astra_db' # 'astra_db'/'local' session = getCQLSession(mode=cqlMode) keyspace = getCQLKeyspace(mode=cqlMode)

In [3]:

message_history = CassandraChatMessageHistory(
    session_id='summarized-conversation-4321',
    session=session,
    keyspace=keyspace,
    ttl_seconds = 3600,
)

message_history = CassandraChatMessageHistory( session_id='summarized-conversation-4321', session=session, keyspace=keyspace, ttl_seconds = 3600, )

In [4]:

message_history.clear()

message_history.clear()

Below is the logic to instantiate the LLM of choice. We chose to leave it in the notebooks for clarity.

In [5]:

import os
from llm_choice import suggestLLMProvider

llmProvider = suggestLLMProvider()
# (Alternatively set llmProvider to 'GCP_VertexAI', 'OpenAI', 'Azure_OpenAI' ... manually if you have credentials)

if llmProvider == 'GCP_VertexAI':
    from langchain.llms import VertexAI
    llm = VertexAI()
    print('LLM from Vertex AI')
elif llmProvider == 'OpenAI':
    os.environ['OPENAI_API_TYPE'] = 'open_ai'
    from langchain.llms import OpenAI
    llm = OpenAI()
    print('LLM from OpenAI')
elif llmProvider == 'Azure_OpenAI':
    os.environ['OPENAI_API_TYPE'] = 'azure'
    os.environ['OPENAI_API_VERSION'] = os.environ['AZURE_OPENAI_API_VERSION']
    os.environ['OPENAI_API_BASE'] = os.environ['AZURE_OPENAI_API_BASE']
    os.environ['OPENAI_API_KEY'] = os.environ['AZURE_OPENAI_API_KEY']
    from langchain.llms import AzureOpenAI
    llm = AzureOpenAI(temperature=0, model_name=os.environ['AZURE_OPENAI_LLM_MODEL'],
                      engine=os.environ['AZURE_OPENAI_LLM_DEPLOYMENT'])
    print('LLM from Azure OpenAI')
else:
    raise ValueError('Unknown LLM provider.')

import os from llm_choice import suggestLLMProvider llmProvider = suggestLLMProvider() # (Alternatively set llmProvider to 'GCP_VertexAI', 'OpenAI', 'Azure_OpenAI' ... manually if you have credentials) if llmProvider == 'GCP_VertexAI': from langchain.llms import VertexAI llm = VertexAI() print('LLM from Vertex AI') elif llmProvider == 'OpenAI': os.environ['OPENAI_API_TYPE'] = 'open_ai' from langchain.llms import OpenAI llm = OpenAI() print('LLM from OpenAI') elif llmProvider == 'Azure_OpenAI': os.environ['OPENAI_API_TYPE'] = 'azure' os.environ['OPENAI_API_VERSION'] = os.environ['AZURE_OPENAI_API_VERSION'] os.environ['OPENAI_API_BASE'] = os.environ['AZURE_OPENAI_API_BASE'] os.environ['OPENAI_API_KEY'] = os.environ['AZURE_OPENAI_API_KEY'] from langchain.llms import AzureOpenAI llm = AzureOpenAI(temperature=0, model_name=os.environ['AZURE_OPENAI_LLM_MODEL'], engine=os.environ['AZURE_OPENAI_LLM_DEPLOYMENT']) print('LLM from Azure OpenAI') else: raise ValueError('Unknown LLM provider.')

LLM from OpenAI

Use the memory with summary in a Chain¶

In [6]:

memory = ConversationSummaryBufferMemory(
    llm=llm,
    chat_memory=message_history,
    max_token_limit=180,
)

memory = ConversationSummaryBufferMemory( llm=llm, chat_memory=message_history, max_token_limit=180, )

In [7]:

summaryConversation = ConversationChain(
    llm=llm, 
    memory=memory,
    verbose=True
)

summaryConversation = ConversationChain( llm=llm, memory=memory, verbose=True )

With the verbose output you can track the summary. It starts empty and once few messages have been exchanged it starts to "incorporate" those messages that would have been lost in the past.

In [8]:

summaryConversation.predict(input='Tell me about William Tell, the hero')

summaryConversation.predict(input='Tell me about William Tell, the hero')

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:

Human: Tell me about William Tell, the hero
AI:

> Finished chain.

Out[8]:

" William Tell is a legendary hero from Swiss folklore. He is most known for his legendary feat of shooting an arrow through an apple perched atop his son's head. According to the legend, William Tell was an expert marksman and was forced to shoot the apple off his son's head to prove his skill. He has become a symbol of Swiss independence and courage."

In [9]:

summaryConversation.predict(
    input="Any relation to the apple from Cupertino?"
)

summaryConversation.predict( input="Any relation to the apple from Cupertino?" )

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary hero from Swiss folklore. He is most known for his legendary feat of shooting an arrow through an apple perched atop his son's head. According to the legend, William Tell was an expert marksman and was forced to shoot the apple off his son's head to prove his skill. He has become a symbol of Swiss independence and courage.
Human: Any relation to the apple from Cupertino?
AI:

> Finished chain.

Out[9]:

' I do not know.'

No messages have slided past the "recent window" yet -- correspondingly, the summary buffer is still empty:

In [10]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[10]:

''

Continue the conversation - you might start to see the summary of past exchanges making its way to the final prompt for the LLM in the form of a "System" contribution to the Current conversation:

In [11]:

summaryConversation.predict(
    input="I heard the two apples may be in fact the very same fruit."
)

summaryConversation.predict( input="I heard the two apples may be in fact the very same fruit." )

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary hero from Swiss folklore. He is most known for his legendary feat of shooting an arrow through an apple perched atop his son's head. According to the legend, William Tell was an expert marksman and was forced to shoot the apple off his son's head to prove his skill. He has become a symbol of Swiss independence and courage.
Human: Any relation to the apple from Cupertino?
AI:  I do not know.
Human: I heard the two apples may be in fact the very same fruit.
AI:

> Finished chain.

Out[11]:

' That is interesting. I cannot confirm if the two apples are indeed the same fruit, as I do not have sufficient information to make that determination.'

This is where the current summary buffer is stored

(Note: depending on the length of the messages so far, at this point you might or might not see an empty summary. In any case, keep reading and it will doubtless start to be nonempty.)

In [12]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[12]:

''

The whole conversation is still in the "recent" window. Below is what will start to happen in the ConversationSummaryBufferMemory at the next chat message: starting from the message list, and the summary so far (if any), the LLM is asked to come up with a new summary which will overwrite the current one, stored in memory.moving_summary_buffer.

In [13]:

print(memory.predict_new_summary(
    memory.chat_memory.messages,
    memory.moving_summary_buffer,
))

print(memory.predict_new_summary( memory.chat_memory.messages, memory.moving_summary_buffer, ))

The human asked the AI about William Tell, the hero from Swiss folklore, and the AI explained that William Tell was known for shooting an arrow through an apple perched atop his son's head and has become a symbol of Swiss independence and courage. The human then asked the AI if the apple from Cupertino is related to the apple in William Tell's story, to which the AI replied that it cannot confirm if the two apples are indeed the same fruit due to lack of information.

In [14]:

summaryConversation.predict(
    input='Ok, enough with apples. Let\'s talk starfish now.'
)

summaryConversation.predict( input='Ok, enough with apples. Let\'s talk starfish now.' )

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary hero from Swiss folklore. He is most known for his legendary feat of shooting an arrow through an apple perched atop his son's head. According to the legend, William Tell was an expert marksman and was forced to shoot the apple off his son's head to prove his skill. He has become a symbol of Swiss independence and courage.
Human: Any relation to the apple from Cupertino?
AI:  I do not know.
Human: I heard the two apples may be in fact the very same fruit.
AI:  That is interesting. I cannot confirm if the two apples are indeed the same fruit, as I do not have sufficient information to make that determination.
Human: Ok, enough with apples. Let's talk starfish now.
AI:

> Finished chain.

Out[14]:

' Sure! Starfish are a type of echinoderm found in marine environments throughout the world. They usually have five arms, but some species can have up to 40. Starfish feed on clams, oysters, and other mollusks, using their tube feet to pry open the shells and their powerful digestive systems to extract the soft parts inside.'

In [15]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[15]:

"\nThe human asks what the AI thinks of artificial intelligence. The AI thinks artificial intelligence is a force for good because it will help humans reach their full potential. The human then asks the AI to tell them about William Tell, the hero. The AI responds that William Tell is a legendary hero from Swiss folklore who is most known for his legendary feat of shooting an arrow through an apple perched atop his son's head. He has become a symbol of Swiss independence and courage."

Now the chat history has presumably reached a certain threshold and there is a summary. In the verbose output below, you will see it appearing in the prompt as a "System" role in the conversation recap.

In [16]:

summaryConversation.predict(
    input='I hear that some of those destroy marine ecosystems.'
)

summaryConversation.predict( input='I hear that some of those destroy marine ecosystems.' )

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
System: 
The human asks what the AI thinks of artificial intelligence. The AI thinks artificial intelligence is a force for good because it will help humans reach their full potential. The human then asks the AI to tell them about William Tell, the hero. The AI responds that William Tell is a legendary hero from Swiss folklore who is most known for his legendary feat of shooting an arrow through an apple perched atop his son's head. He has become a symbol of Swiss independence and courage.
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary hero from Swiss folklore. He is most known for his legendary feat of shooting an arrow through an apple perched atop his son's head. According to the legend, William Tell was an expert marksman and was forced to shoot the apple off his son's head to prove his skill. He has become a symbol of Swiss independence and courage.
Human: Any relation to the apple from Cupertino?
AI:  I do not know.
Human: I heard the two apples may be in fact the very same fruit.
AI:  That is interesting. I cannot confirm if the two apples are indeed the same fruit, as I do not have sufficient information to make that determination.
Human: Ok, enough with apples. Let's talk starfish now.
AI:  Sure! Starfish are a type of echinoderm found in marine environments throughout the world. They usually have five arms, but some species can have up to 40. Starfish feed on clams, oysters, and other mollusks, using their tube feet to pry open the shells and their powerful digestive systems to extract the soft parts inside.
Human: I hear that some of those destroy marine ecosystems.
AI:

> Finished chain.

Out[16]:

'  Yes, some starfish, such as the Crown-of-Thorns starfish, can cause significant damage to coral reefs and other marine ecosystems if their populations become too large. They feed on coral polyps and can quickly decimate large areas of reef. Management strategies such as culling and breeding programs have been implemented to help reduce their impact.'

In [17]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[17]:

"\nThe human asks what the AI thinks of artificial intelligence. The AI thinks artificial intelligence is a force for good because it will help humans reach their full potential. The human then asks the AI to tell them about William Tell, the hero. The AI responds that William Tell is a legendary hero from Swiss folklore who is most known for his legendary feat of shooting an arrow through an apple perched atop his son's head. He has become a symbol of Swiss independence and courage. The human then asks if there is any relation between William Tell's apple and the Apple from Cupertino and the AI responds that it cannot confirm if the two apples are the same due to the lack of sufficient information."

From this point on, the summary will not grow indefinitely in size, but will rather try to cover a longer and longer conversation in a fixed amount of text.

In [18]:

print(memory.predict_new_summary(
    memory.chat_memory.messages,
    memory.moving_summary_buffer,
))

print(memory.predict_new_summary( memory.chat_memory.messages, memory.moving_summary_buffer, ))

The human asks what the AI thinks of artificial intelligence. The AI thinks artificial intelligence is a force for good because it will help humans reach their full potential. The human then asks the AI to tell them about William Tell, the hero and the AI responds that he is a legendary hero from Swiss folklore who is most known for his legendary feat of shooting an arrow through an apple perched atop his son's head. The human then asks if there is any relation between William Tell's apple and the Apple from Cupertino and the AI responds that it cannot confirm if the two apples are the same due to the lack of sufficient information. The human then changes the topic to starfish and the AI explains that starfish are a type of echinoderm found in marine environments throughout the world and some of them can cause significant damage to coral reefs and other marine ecosystems if their populations become too large. Management strategies such as culling and breeding programs have been implemented to help reduce their impact.