I get the error "'text': 'dict' object cannot be converted to 'PyString' error" while composing an LCEL pipeline

[lorenzobalzani]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

@staticmethod
    def _load_custom_prompt(prompt_path, rag_top_k):
        """
        Load the custom prompt from the prompt path, both the system and user messages.

        Args:
            prompt_path (str): Path to the prompt file
            rag_top_k (int): Number of documents to retrieve

        Returns:
            ChatPromptTemplate: the chat prompt template, with the system and user messages.
        """
        with open(prompt_path, "r", encoding="utf-8") as f:
            prompts = json.loads(f.read())

        return ChatPromptTemplate.from_messages(
            [
                ("system", prompts["system_message"].format(k=rag_top_k, context="{context}")),
                MessagesPlaceholder("chat_history"),
                ("human", prompts["user_message"]),
            ]
        )

llm = AzureChatOpenAI(deployment_name=deployment_name, temperature=temperature, verbose=True)

            retriever = self._vector_store.as_retriever(
                search_type="similarity_score_threshold",
                search_kwargs={
                    "k": rag_top_k,
                    "score_threshold": rag_score_threshold,
                    "filter": rag_filter or {},
                })

            prompt = "hi, you have a context of documents {context} to respond to this input: {input}."

            question_answer_chain = prompt | llm | StrOutputParser()

            rag_chain = RunnableParallel({
                "context": retriever,  # Automatically applies the retriever.
                "input": RunnablePassthrough(),  # Pass the question directly.
                "chat_history": RunnablePassthrough()  # Ensure chat history is passed through as well.
            }) | question_answer_chain

            chat_history = [message for interaction in chat_history for message in interaction]
            eval_recorder = self._evaluator.get_chain_recorder(rag_chain, self._collection_name, self._cheapest_llm)
            with get_openai_callback() as openai_callback, eval_recorder:
                ai_answer = rag_chain.invoke({"input": user_input, "chat_history": chat_history})
            logger.info(openai_callback)

Description

I get that error (contained in the title) when trying to append the chat_history while calling the invoke method.

Am I wrongly composing the pipeline? Where should I put the chat_history?

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 23.5.0: Wed May 1 20:14:59 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T8122
Python Version: 3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 10:07:17) [Clang 14.0.6 ]

Package Information

langchain_core: 0.2.26
langchain: 0.2.11
langchain_community: 0.2.10
langsmith: 0.1.95
langchain_openai: 0.1.20
langchain_text_splitters: 0.2.2

[dosubot[bot]]

To avoid the "'text': 'dict' object cannot be converted to 'PyString'" error, you should ensure that the chat_history is correctly placed in the prompt template and passed as a string. Here is the correct way to compose the LCEL pipeline:

from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_openai import ChatOpenAI

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a pirate. Answer the following questions as best you can."),
        ("placeholder", "{chat_history}"),
        ("human", "{input}"),
    ]
)

history = InMemoryChatMessageHistory()

def get_history():
    return history

chain = prompt | ChatOpenAI() | StrOutputParser()

wrapped_chain = RunnableWithMessageHistory(
    chain,
    get_history,
    history_messages_key="chat_history",
)

wrapped_chain.invoke({"input": "how are you?"})

In this setup, the chat_history is correctly referenced in the prompt template and passed through the RunnableWithMessageHistory configuration. This should prevent the error related to converting a 'dict' object to 'PyString' ^[1][2].

Additionally, you can refer to the LangChain documentation for more detailed guidance on managing chat history in conversational applications. The documentation provides examples of how to use BaseChatMessageHistory and RunnableWithMessageHistory to manage chat history effectively ^[1][2].

To continue talking to Dosu, mention @dosu.

[travisday]

I am having this same issue. It seems to only appear when using a retriever with multiple input parameters as runnable passthroughs.

This chain DOES NOT work

rag_chain = (
        {
            "context": retriever | format_docs,
            "fitness_level": RunnablePassthrough(),
            "goals": RunnablePassthrough(),
        }
        | prompt
        | llm
        | StrOutputParser()
    )

    workout = rag_chain.invoke({"fitness_level": fitness_level, "goals": goals})

But it does work if I remove the retriever.

langchain = "0.3.0"

[lorenzobalzani]

From which point in the code do you remove the retriever?

[ehsanahmadkhan]

I has the same issue I made a custom solution that worked for me use this
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

db = FAISS.load_local(file_path, embeddings, allow_dangerous_deserialization=True)

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

prompt_response = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            """Based on the question, and user history, write a natural language response that will answer the customer query:  
                           context: {context}
            """,
        ),

        ("human", "{question}"),
    ]
)

llm = ChatOpenAI(model=gpt_model, temperature=0)
full_chain = (
    RunnablePassthrough.assign(
        context=lambda x: format_docs(db.similarity_search(x["question"], k=3)),
   
    )
    | prompt_response
    | llm
    | StrOutputParser()
)

[lorenzobalzani]

@dosu I stille get the same error

[dosubot[bot]]

The issue you're encountering is likely due to the way the RunnablePassthrough is handling the input parameters. When using multiple input parameters with a retriever, you need to ensure that the data is correctly formatted and passed through the pipeline.

Here's a revised version of your chain setup that should work:

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnableParallel, RunnablePassthrough
from langchain_openai import ChatOpenAI

# Define your prompt template
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a fitness assistant. Answer the following questions based on the context."),
        ("human", "{fitness_level} {goals}"),
    ]
)

# Define your language model
llm = ChatOpenAI()

# Define your retriever and format_docs function
# Assuming format_docs is a function that formats the retrieved documents
def format_docs(docs):
    return "\n".join([doc.page_content for doc in docs])

# Define the parallel runnable
rag_chain = RunnableParallel({
    "context": retriever | format_docs,
    "fitness_level": RunnablePassthrough(),
    "goals": RunnablePassthrough(),
}) | prompt | llm | StrOutputParser()

# Invoke the chain with the input parameters
workout = rag_chain.invoke({"fitness_level": fitness_level, "goals": goals})

In this setup:

1. The RunnableParallel is used to handle multiple input parameters.
2. The retriever is combined with the format_docs function to format the retrieved documents.
3. The RunnablePassthrough is used to pass the fitness_level and goals parameters directly.
4. The ChatPromptTemplate is structured to include the fitness_level and goals in the prompt.

This should resolve the issue with the 'text': 'dict' object cannot be converted to 'PyString' error by ensuring that the input parameters are correctly passed through the pipeline ^[1][2].

To continue talking to Dosu, mention @dosu.

[mingjun1120]

I'm experiencing the same error. Does anyone know how to fix it?

[ehsanahmadkhan525]

I had the same issue I made a custom solution that worked for me use this
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

db = FAISS.load_local(file_path, embeddings, allow_dangerous_deserialization=True)

def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)

prompt_response = ChatPromptTemplate.from_messages(
[
(
"system",
"""Based on the question, and user history, write a natural language response that will answer the customer query:
context: {context}
""",
),

    ("human", "{question}"),
]

)

llm = ChatOpenAI(model=gpt_model, temperature=0)
full_chain = (
RunnablePassthrough.assign(
context=lambda x: format_docs(db.similarity_search(x["question"], k=3)),

)
| prompt_response
| llm
| StrOutputParser()

)

[rpetar]

This should work:

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough, chain

model = None
retriever = None

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a chatbot. Answer the following questions based on the context."),
        ("human", "Based on this context:{context}\nRespond to this question: {question}"),
    ]
)

def format_docs(docs):
    """Format the retrieved documents into a single string."""
    return "\n".join([doc.page_content for doc in docs])

@chain
def run_retriever(input_):
    """Run the retriever to get the relevant documents."""
    docs = retriever.invoke(input_["context_question"])
    return format_docs(docs)

rag_chain = RunnablePassthrough.assign(context=run_retriever) | prompt | model | StrOutputParser()

output = rag_chain.invoke({"context_question": "What is RAG?", "question": "What is the context about?"})